Running a benchmark with multiple experiments results in similar calls to the utility object. In between experiments one has to decide if the cache needs to be emptied or not.
Despite the same samples CWS can't use the same cache as TMC. Abstractly speaking it might happen that there are multiple experiments E1, E2, E3. Assume the transition form E1 to E2 invalidates the cache, but E3 can reuse the cache. By imposing the algorithmic knowledge when to invalidate caches on the user we disable the access to certain features to others. It would be favorable if there are multiple experiment caches. These high level caches might use a LRU(10) policy (the sub objects might use a LRU(10000) policy).
Verifying, that the same cache can be reused from a historic experiment EH, can be done with a signature of the utility object. As the value of the utility depends on the model, dataset and scorer. It also depends on the valuation method in our current state (CWS imposes a modified scorer), but this would be caught by estimating the equivalence of the scorer in the signature.
So my proposal would be to add:
[ ] Calculate signature of utility function to be used for caching
[ ] (Optional) Signature is a premature check and might be improved by using statistic sampling of input and output values
[ ] Pass two numbers, n_low_level and n_high_level for specifying size of both LRU policies.
Running a benchmark with multiple experiments results in similar calls to the utility object. In between experiments one has to decide if the cache needs to be emptied or not.
Despite the same samples CWS can't use the same cache as TMC. Abstractly speaking it might happen that there are multiple experiments
E1,E2,E3. Assume the transition formE1toE2invalidates the cache, butE3can reuse the cache. By imposing the algorithmic knowledge when to invalidate caches on the user we disable the access to certain features to others. It would be favorable if there are multiple experiment caches. These high level caches might use aLRU(10)policy (the sub objects might use aLRU(10000)policy).Verifying, that the same cache can be reused from a historic experiment
EH, can be done with a signature of the utility object. As the value of the utility depends on the model, dataset and scorer. It also depends on the valuation method in our current state (CWS imposes a modified scorer), but this would be caught by estimating the equivalence of the scorer in the signature.So my proposal would be to add:
n_low_levelandn_high_levelfor specifying size of bothLRUpolicies.