ckan / datapusher

A standalone web service that pushes data files from a CKAN site resources into its DataStore
GNU Affero General Public License v3.0
77 stars 155 forks source link

Options to avoid creating stale DataStore content for external data resources #54

Open gjlawran opened 10 years ago

gjlawran commented 10 years ago

Because data resources added as a path to an external data source (i.e. a file that is not uploaded to the FileStore) are subject to update, the content of the DataStore can go stale. If services are built off the DataStore API they will be using out-of-date data - as will be the previews.

Considering turning off default / automatic DataPusher additions for data resources added as a path to a data resource (e.g. externally hosted file) . Would still allow manual DataPusher additions.

Also considering developing mechanism to regularly monitor currency of external data resources to determine when DataStore update via DataPusher is required.