Remote resources are not supported

[hhkaos]

I have register the rwlabs host via:

curl --data "host=https://data.hdx.rwlabs.org&id=rwlabs" localhost:1337/ckan

But then when I try to access to this dataset:

http://localhost:1337/ckan/rwlabs/ourairports-ind

I'm getting this error:

Unable to retrieve data from https://data.hdx.rwlabs.org/datastore/dump/45771200-34c9-4d0d-9a88-7c58512af19b

I checked the code and ckan.ckan_dump_path at line 38 of ckan.js and it seems to point the right path, but it returns a 404. Do you know how to solve this?

Thanks! Raul P.S: I have also been trying to do the same with this dataset via http://localhost:1337/ckan/fiware/control-de-plagas-para-el-ao-2015 and getting the same error

[hhkaos]

I've just noticed that you are also talking about this issue here, sorry!

I'll check myself and try to solve it too (wish me luck xD)

[chelm]

@dmfenton do you have insight into this issue? Why is the "dump" url returning 404

[dmfenton]

So Koop-CKAN only handles datasets that are stored in the CKAN datastore. The dataset you are trying to access is a remote resource: http://ourairports.com/countries/IN/airports.csv

I recommend we rewrite this issue as a feature request to handle remote CSVs

[dmfenton]

Here's the JSON view of this dataset for future work on this curl -XGET 'https://data.hdx.rwlabs.org/api/3/action/package_show?id=ourairports-ind'

{
  "help": "Return the metadata of a dataset (package) and its resources.\n\n    :param id: the id or name of the dataset\n    :type id: string\n    :param use_default_schema: use default package schema instead of\n        a custom schema defined with an IDatasetForm plugin (default: False)\n    :type use_default_schema: bool\n\n    :rtype: dictionary\n\n    ",
  "success": true,
  "result": {
    "license_title": "other-pd-nr",
    "maintainer": "OurAirports",
    "relationships_as_object": [],
    "package_creator": "david",
    "private": false,
    "maintainer_email": "contact@ourairports.com",
    "num_tags": 3,
    "id": "5645d365-f8ec-434b-99ad-fc738d40b0de",
    "metadata_created": "2014-06-02T20:21:21.055677",
    "caveats": "Unverified live data. May change at any time.",
    "metadata_modified": "2014-09-24T18:33:59.279561",
    "author": null,
    "author_email": null,
    "state": "active",
    "methodology": "Other",
    "version": null,
    "dataset_source": "OurAirports.com community web site.",
    "license_id": "other-pd-nr",
    "type": "dataset",
    "resources": [
      {
        "resource_group_id": "c91f340a-1731-4926-931b-7e9a27a53516",
        "cache_last_updated": null,
        "revision_timestamp": "2014-09-24T18:33:59.278075",
        "webstore_last_updated": null,
        "datastore_active": false,
        "id": "45771200-34c9-4d0d-9a88-7c58512af19b",
        "size": null,
        "state": "active",
        "hash": "",
        "description": "Spreadsheet listing airport locations in India.  Unverified community data from ourairports.com.  Note that this data comes live from the web site, and can change at any time.",
        "format": "CSV",
        "tracking_summary": {
          "total": 13,
          "recent": 4
        },
        "last_modified": null,
        "url_type": null,
        "mimetype": "text/csv",
        "cache_url": null,
        "name": "List of airports in India",
        "created": "2014-09-24T18:33:59.339513",
        "url": "http://ourairports.com/countries/IN/airports.csv",
        "webstore_url": null,
        "mimetype_inner": null,
        "position": 0,
        "revision_id": "bebc89f3-c4ce-4d19-bb54-e54402680de7",
        "resource_type": null
      }
    ],
    "num_resources": 1,
    "tags": [
      {
        "vocabulary_id": null,
        "display_name": "aviation",
        "name": "aviation",
        "revision_timestamp": "2014-06-02T20:21:21.055677",
        "state": "active",
        "id": "4163ef25-1481-45ec-89e6-a194584ce51c"
      },
      {
        "vocabulary_id": null,
        "display_name": "geodata",
        "name": "geodata",
        "revision_timestamp": "2014-06-02T20:21:21.055677",
        "state": "active",
        "id": "29e4170e-6340-4949-b2ae-3c42062d012e"
      },
      {
        "vocabulary_id": null,
        "display_name": "transportation",
        "name": "transportation",
        "revision_timestamp": "2014-06-02T20:21:21.055677",
        "state": "active",
        "id": "6098ee16-6779-4819-82f4-79b82d55f114"
      }
    ],
    "tracking_summary": {
      "total": 66,
      "recent": 9
    },
    "groups": [
      {
        "display_name": "India",
        "description": "",
        "image_display_url": "",
        "title": "India",
        "id": "ind",
        "name": "ind"
      }
    ],
    "creator_user_id": null,
    "relationships_as_subject": [],
    "revision_timestamp": "2014-09-24T18:33:59.278075",
    "organization": {
      "description": "Collaborative site for collecting and sharing open global aviation data.\r\n\r\nhttp://ourairports.com",
      "created": "2014-04-24T22:00:54.948536",
      "title": "OurAirports",
      "name": "ourairports",
      "revision_timestamp": "2014-04-24T22:02:17.577484",
      "is_organization": true,
      "state": "active",
      "image_url": "",
      "revision_id": "720eae06-2877-4d13-8af4-30061f6a72a5",
      "type": "organization",
      "id": "dc65f72e-ba98-40aa-ad32-bec0ed1e10a2",
      "approval_status": "approved"
    },
    "name": "ourairports-ind",
    "isopen": false,
    "url": "http://ourairports.com/countries/IN/",
    "notes": "List of airports in India, with latitude and longitude.  Unverified community data from ourairports.com.",
    "owner_org": "dc65f72e-ba98-40aa-ad32-bec0ed1e10a2",
    "title": "Airports in India",
    "revision_id": "bebc89f3-c4ce-4d19-bb54-e54402680de7"
  }
}

[chelm]

Worth mentioning that this did work at one point, hence the example URL pointing to the reference.

Somewhere along the way we changed the support for the remove files. I'll re-add it.

[dmfenton]

Are you sure it wasn't the CKAN dataset that changed?

[dmfenton]

Ah actually, just looking at the first commit, we did support remote CSVs

[chelm]

@dmfenton yes. I have an original version of the provider and it works great.

[chelm]

While we're at it this needs better tests, etc. So I can spend some time add this and bringing up to a 1.0.0 release pre-UC

[dmfenton]

Here's the commit: https://github.com/koopjs/koop-ckan/commit/435ea4f0ceb96e3c56be4de04752b9dc16c8d75e where it was broken

[chelm]

@dmfenton do you by chance have ckan dataset that the new dump path thing?

[dmfenton]

http://demo.ckan.org/datastore/dump/5a643af5-af8b-4db6-b2b0-23910245ddce

[hhkaos]

Hi guys!, could you make any progress in here?