search

SWS-Methodology / faoswsUtil

This is an add-on utilities package to the faosws package
0 stars 0 forks source link

Add caching for conversion functions #26

Open sebastian-c opened 8 years ago

sebastian-c commented 8 years ago

Some conversion functions like fcl2cpc use the same table every time and should be at least optionally cached so as not to repeatedly download the file.

sebastian-c commented 6 years ago

The easiest way to do this caching would be to have a memoised version. The function would cache in its local environment. This works because the tables are small, but if it's used hundreds of times, it represents a real saving.

Should caching be opt-in ot opt-out?

chrMongeau commented 6 years ago

Would the memoised version ensure that if the original datatable changes, the local version will "refresh" (update to the remote version)?

sebastian-c commented 6 years ago

It would read the table the first time it is called, i.e. it resets when the package is reloaded. So in practice, the table would be refreshed with each module run. I'm tempted to go with caching by default with an option to force refresh the cache.

In addtion, there could be an options(faosws.enableCache) = TRUE which would allow disabling of the cache altogether so it doesn't have to be done for all function calls.

chrMongeau commented 6 years ago

Oh, OK. So the caching would would inside a module run, i.e., if the same function that uses a remote datatable is called more than once in a run, it will use the cached version. It would'n work in separate runs, as I suggested in https://github.com/SWS-Methodology/faoswsTrade/issues/155 ? (Maybe in that case the mechanism shoud be ad hoc as the remote datatable is very big.)

In any case, caching seems, IMO, a very good idea.

sebastian-c commented 6 years ago

The trick with your caching problem is that you need somewhere to store the data. If fread is a lot faster than ReadDatatable, then you need a table that has two dates. One that has the last time the source table was updated and another representing the last time your local data was updated. When the the source data is newer than the local, download the new data.

From: Christian Mongeau notifications@github.com Sent: Wednesday, 15 November 2017 11:01:11 AM To: SWS-Methodology/faoswsUtil Cc: Campbell, Sebastian (ESS); Author Subject: Re: [SWS-Methodology/faoswsUtil] Add caching for conversion functions (#26)

Oh, OK. So the caching would would inside a module run, i.e., if the same function that uses a remote datatable is called more than once in a run, it will use the cached version. It would'n work in separate runs, as I suggested in SWS-Methodology/faoswsTrade#155https://github.com/SWS-Methodology/faoswsTrade/issues/155 ? (Maybe in that case the mechanism shoud be ad hoc as the remote datatable is very big.)

In any case, caching seems, IMO, a very good idea.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/SWS-Methodology/faoswsUtil/issues/26#issuecomment-344543930, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABu_1YAFgQBbSfe_ykA0CP8TF_EVrdNUks5s2rZmgaJpZM4KHlkY.