Import a Data Package JSON to CKAN

[Stephen-Gates]

See acceptance test

Ensure that the existing functionality imports all the metadata for a Data Package into CKAN.

Valid test data:

• https://github.com/frictionlessdata/example-data-packages/blob/master/world-continents-topojson/datapackage.json

Invalid test data:

• https://raw.githubusercontent.com/ODIQueensland/ckan-data-curator-integration/master/test/data/invalid-data-packages/geo-lat-lon-as-numbers/datapackage.json
• https://github.com/frictionlessdata/example-data-packages/blob/master/geo-countries/datapackage.json data is not hosted elsewhere

[ghost]

creating a dummy organization (which is then available for selection on download), and uploading valid datapackage.json

{
  "name": "world-continents-topojson",
  "title": "World Continents TopoJSON Example Data Package",
  "licenses": [{
    "title": "MIT License",
    "path": "http://www.opensource.org/licenses/MIT"
  }],
  "sources": [{
    "title": "TopoJSON Collection - World Continents",
    "path": "https://github.com/deldersveld/topojson/blob/master/world-continents.json"
  }],
  "resources": [{
    "path": "https://github.com/deldersveld/topojson/raw/master/world-continents.json",
    "name": "world-continents",
    "description": "World Continents",
    "format": "topojson",
    "mediatype": "application/json"
  }]
}

This record then, on download from ckan, becomes:

{
  "extras": {
    "licenses": [
      {
        "path": "http://www.opensource.org/licenses/MIT", 
        "title": "MIT License"
      }
    ], 
    "profile": "data-package"
  }, 
  "name": "world-continents-topojson", 
  "resources": [
    {
      "path": "https://github.com/deldersveld/topojson/raw/master/world-continents.json", 
      "title": "world-continents", 
      "description": "World Continents", 
      "name": "world-continents", 
      "format": "topojson"
    }
  ], 
  "title": "World Continents TopoJSON Example Data Package"
}

[ghost]

Hi @Stephen-Gates Looking at mapping, I realise that it's in kindofa draft state, but how much of that is expected to be done in this sprint?

[ghost]

And:

{
  "name": "geo-countries",
  "title": "Country Polygons as GeoJSON (Example)",
  "description": "Example Data Package providing GeoJSON polygons for all the world's countries",
  "licenses": [{
    "id": "CC0-1.0",
    "title": "CC0 1.0",
    "url": "https://creativecommons.org/publicdomain/zero/1.0/"
  }],
  "resources": [
    {
      "name": "countries",
      "path": "data/countries.geojson",
      "format": "geojson",
      "mediatype": "application/json",
      "schema": {
        "fields": [
          {
            "name": "ADMIN",
            "description": "Common name of the country",
            "type": "string"
          },
          {
            "name": "ISO_A3",
            "description": "3 characters code for the country, according to ISO3166 standard",
            "type": "string"
          }
        ]
      }
    }
  ]
}

becomes:

{
  "description": "Example Data Package providing GeoJSON polygons for all the world's countries", 
  "extras": {
    "licenses": [
      {
        "url": "https://creativecommons.org/publicdomain/zero/1.0/", 
        "id": "CC0-1.0", 
        "title": "CC0 1.0"
      }
    ], 
    "profile": "data-package"
  }, 
  "name": "geo-countries", 
  "resources": [
    {
      "path": "https://raw.githubusercontent.com/frictionlessdata/example-data-packages/master/geo-countries/data/countries.geojson", 
      "schema": "{u'fields': [{u'type': u'string', u'name': u'ADMIN', u'description': u'Common name of the country'}, {u'type': u'string', u'name': u'ISO_A3', u'description': u'3 characters code for the country, according to ISO3166 standard'}]}", 
      "title": "countries", 
      "name": "countries", 
      "format": "GeoJSON"
    }
  ], 
  "title": "Country Polygons as GeoJSON (Example)"
}

[ghost]

@Stephen-Gates Any chance we have more comprehensive examples than these? Trying to make sense of mapping is difficult with only these tests to go by? Perhaps we're not be there yet?, but would be good to show in github issue, what a comprehensive example becomes atm, then show the expectation of what this should be in the acceptance test.

[Stephen-Gates]

I think scope should be a least fix mappings in https://github.com/ODIQueensland/ckan-data-curator-integration/issues/2

I’ll look deeper

[Stephen-Gates]

Creating an organisation is done manually in the user interface and not created by this feature. The organisation is needed to add data and assumed to pre-exist

[Stephen-Gates]

See https://github.com/frictionlessdata/ckanext-datapackager/issues/62#issuecomment-364224790 for licenses.

So in your first example there wouldn't be extras in the download, just the standard licences from the spec.

[ghost]

Hi @Stephen-Gates Just put them there for reference for work on this ticket - what would be good would be to have examples that have more than just licenses.

[Stephen-Gates]

Just created test datapackage with more properties https://raw.githubusercontent.com/ODIQueensland/ckan-data-curator-integration/da2f80af24509aece042884412ca22fac13ecf55/test/data/valid-data-packages/cpi-data-via-url/datapackage.json

Uploading to CKAN using data-packager extension