the created datapackage should conform to a version of frictionlessdata specs

hasadna / knesset-data-datapackage

Provides downloadable Knesset (Israeli parliament) data

1 stars 5 forks source link

the created datapackage should conform to a version of frictionlessdata specs #15

Open OriHoch opened 7 years ago

OriHoch commented 7 years ago

reproduction steps

try to open / use the datapackage from a frictionlessdata compatible tool

expected

should work

actual

doesn't work with some or all tools
need to ensure we conform to the specs or use the datapackage tools to create the datapackage
need to decide which version of the specs we want to support

OriHoch commented 7 years ago

@akariv @ydaniv - if you had specific problems you encountered, it would be helpful if you could describe them here

OriHoch commented 7 years ago

I think we did conform to a spec at one point, but possible either the specs changed or our datapackage changed..

should add a unit test as well to ensure we will conform in the future too

OriHoch commented 7 years ago

need to bear in mind that frictionlessdata specs are still not finalized fully so, when choosing version for spec conforming, your options are :

conform to latest pre-v1 version
conform to a specific v1 pre release
conform to latest specs

each option has pros and cons..

OriHoch commented 7 years ago

related issue: #14

akariv commented 7 years ago

I think that until v1 specs are finalized, there's no point in trying to conform to them (they keep changing all the time).

The test should simply be to

open the datapackage using datapackage.DataPackage() and make sure it validates.
fully iterate on all resources to make sure the data is valid.

OriHoch commented 7 years ago

~~I tried to do it but our code is Python 2.7 (for compatibility with old Open Knesset code)~~

~~opened an issue to have the low level library python 3 compatible - hasadna/knesset-data-python#19~~

commented on wrong issue..

akariv commented 7 years ago

@OriHoch, datapackage-py is 2.7 compatible: https://travis-ci.org/frictionlessdata/datapackage-py/jobs/235951075

OriHoch commented 7 years ago

oops, commented on wrong issue, I was referring to datapackage-pipelines - #14

ydaniv commented 7 years ago

Issues we encountered with committees:

path doesn't conform to spec. It's generated as a list. Specifically in committees-meetings-protocols.
datetime is generated in a naive format and missing a z. We fixed it by adding a format property according to the generated format.

Also: missing schema for the protocol files, so resources aren't generated for protocols.

OriHoch commented 7 years ago

@ydaniv regarding the protocols - which datapackage did you use? the datapackage for last 5 days doesn't have protocols because it takes more then 5 days for Knesset to upload protocol for a meeting.

You should get a datapackage for last 120 days if you need the protocols (you can see in oknesset-travis channel in slack, every week there is a package for last 120 days)