datapackages with invalid date/time formats should be dumped and loaded successfully

datahq / dataflows

DataFlows is a simple, intuitive lightweight framework for building data processing flows in python.

https://dataflows.org

MIT License

195 stars 40 forks source link

datapackages with invalid date/time formats should be dumped and loaded successfully #41

Closed OriHoch closed 5 years ago

OriHoch commented 5 years ago

Seems there is a problem with datapackages created with the non-standard date/time format

The packages are loaded fine but are saved incorrectly and can't be loaded

This affects the checkpoint as it depends on dump_to_path and load - which makes it "misbehave" in some cases

see example here - https://gist.github.com/OriHoch/5e5b608f31916f27beafb40376860761

akariv commented 5 years ago

I'm guessing the problem is that dump_to_path modifies the date format on save but does not update the datapackage, so a discrepancy is created. I'll handle this tomorrow.

akariv commented 5 years ago

So tomorrow was a bit optimistic... should be fixed via https://github.com/datahq/dataflows/pull/52