search

talos / opendatacache

caching and compression for socrata open data portals
23 stars 3 forks source link

Revisions #2

Open jqnatividad opened 9 years ago

jqnatividad commented 9 years ago

It'd be nice if snapshots of the data are also kept around. So we can turn back the clock and fetch the dataset for a given date. Most of these datasets are not really that big.

Also, S3 storage is getting cheaper by the day, and with Glacier, we can even move datasets off to long-term storage if a certain quota is set.

This will essentially make opendatacache an opendatatimemachine as well :smile:

talos commented 9 years ago

Socrata does keep a few old versions around right now, although the interface to pull those old ones is not transparent at the moment.

I have some hope that dat will eventually be a scalable alternative to keeping dozens of big CSVs lying around. But as of several months ago I did not find the performance suitable...

jqnatividad commented 9 years ago

Spoke with @karissa last week. With dat beta officially out, it might be worthwhile to revisit as we're investigating how use dat with CKAN as well.