SWS-Methodology / faoswsTrade

World trade data processing for the FAO Statistical Working System
http://www.fao.org/economic/ess/ess-home/en/
5 stars 2 forks source link

Trade datatables caching #155

Open chrMongeau opened 7 years ago

chrMongeau commented 7 years ago

The module spends around half an hour downloading both the esdata and tldata datatables. There should be a way to use a caching mechanisms that uses either a hash of the whole table or an available date of last update. The idea is that the module should check whether the hash/date changed (which implies that the dataset changed) and proceed with the download only if it did; if the hash/date didn't change the module should load some locally available corresponding datatable. I was told that the "last update date" can be made available as a datatable, so that the module use it could use it to check if a download is necesssary).

chrMongeau commented 6 years ago

On 2017-11-10 I sent an email mentioning this feature and also proposing a new one (delete non-existing observations server-side).