Check modified date and hash in datastore_upload

ckan / ckanext-datastorer

Get files from ckan into the webstore.

21 stars 18 forks source link

Closed johnglover closed 11 years ago

johnglover commented 11 years ago

The datastore_upload command should try to avoid downloading and datastoring resources unless they have changed. It could:

do a HEAD request and (if accepted) see if there is a last-modified header, check that it is more recent than the resource's last modified date
after downloading check to see if the hash has changed before datastoring

nigelbabu commented 11 years ago

r+ significantly reduced time to do an import.