Geo data packages are throwing 500 Internal Server Error

[zelima]

Following packages have problems after publishing:

https://staging.datapackaged.com/core/geo-nuts-administrative-boundaries | status code: 500 https://staging.datapackaged.com/core/geo-countries | status code: 500 https://staging.datapackaged.com/core/geo-admin1-us | status code: 500 https://staging.datapackaged.com/core/geo-ne-admin1 | status code: 500

heroku logs

For full logs see heroku logs

2017-04-10T05:56:56.804332+00:00 app[web.1]: Traceback (most recent call last):
...

...
2017-04-10T05:56:56.804362+00:00 app[web.1]:   File "/app/app/templates/_snippets.html", line 55, in template
2017-04-10T05:56:56.804364+00:00 app[web.1]:     <i class="icon-legal"></i> <a href="{{dataset.licenses[0].url}}" title="Available under the following License">{{dataset.licenses[0].name or dataset.licenses[0].id}}</a>
2017-04-10T05:56:56.804365+00:00 app[web.1]:     return getattr(obj, attribute)
2017-04-10T05:56:56.804364+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/jinja2/environment.py", line 430, in getattr
2017-04-10T05:56:56.804365+00:00 app[web.1]: UndefinedError: dict object has no element 0

Acceptance criteria

• [x] Non of datapckages above has 500 errors

Tasks

• [x] determine why this is working for other data packages.
  • do other ones have license?
  • if yes, why geo datapackages do not have licenses, is it coinsidence
• [x] decide which option is more acceptable
  • modify datapackage.json inside all 4 repos for now
• [x] implement

Analysis

Exploring the code further discovered that all 4 Data Packages have common issue:

Inside datapackage.json all of them have property licenses that are set as objects Eg https://github.com/datasets/geo-nuts-administrative-boundaries/blob/master/datapackage.json#L20

"licenses": {
    "type": "ODC-PDDL",
    "url": "http://opendatacommons.org/licenses/pddl/1.0/"
  },

Following the https://specs.frictionlessdata.io description of required and optional properties for a Data Package descriptor https://specs.frictionlessdata.io/data-package/#properties property licenses should be list of objects

licenses The license(s) under which the resource is published. This property is not legally binding and does not guarantee that the package is licensed under the terms defined herein. Items Each item in the Licenses array is a License object. The uri property is required, and other defined properties are optional.

We have several options:

• have the flag inside jinja templates and check for type.
  • too dirty, Plus what if user decides it should be another type Eg string
• Modify on server side - have helper function that checks if it is list and appends to list if not
  • again what if other types?
• Modify the validation script, so that it throws warning/error if licenses is not set according to specs. This way we are guaranteed that it will always be list of the objects
  • one more validation error for user. Creating data packages may become too complex, while it should be simple and straightforward
• Modify actual datapackage.jsons inside repos

As a quick and simple solution for this 4 packages we decided to manually update datapckage.json and rise pull requests

Pull requests:

• [x] geo-countries - https://github.com/datasets/geo-countries/pull/9
• [x] geo-ne-admin1 - https://github.com/datasets/geo-ne-admin1/pull/2
• [x] geo-ne-admin1-us -https://github.com/datasets/geo-ne-admin1-us/pull/2
• [x] geo-nuts-administrative-boundaries - https://github.com/datasets/geo-nuts-administrative-boundaries/pull/3

[rufuspollock]

@zelima what about our discussion about the fact this was causing a 500 error. Whilst we can fix this immediate issue by fixing the data packages what about similar issues in future. We need to think about where in the system we handle "bad" data packages.

[zelima]

FIXED all 4 Data Packages are live and working fine