search

ben-denham / labtech

Easily run experiment permutations with multi-processing and caching
https://ben-denham.github.io/labtech/
GNU General Public License v3.0
7 stars 1 forks source link

Clear cache when `run` function changes. #26

Open nathanjmcdougall opened 3 months ago

nathanjmcdougall commented 3 months ago

Often the run function is being modified in between runs, which invalidates the cache.

Dagster copes with this by having the user maintain a hard coded version in the decorator: https://docs.dagster.io/guides/dagster/asset-versioning-and-caching

I think labtech could do something similar, to enable automatic cache invalidation.

ben-denham commented 3 months ago

I've just been thinking about whether there was anything to be done about code versioning, and this looks like a great idea, thanks!

Should be straightforward to implement without breaking existing caches.

I think an analogous argument to the task decorator would be good, and it would keep consistency between the tools.

I considered whether hashing the task's code would work, but you'd run into issues with changes that don't affect output and with dependency code.

On Mon, 29 July 2024, 11:35 Nathan McDougall, @.***> wrote:

Often the run function is being modified in between runs, which invalidates the cache.

Dagster copes with this by having the user maintain a hard coded version in the decorator: https://docs.dagster.io/guides/dagster/asset-versioning-and-caching

I think labtech could do something similar, to enable automatic cache invalidation.

— Reply to this email directly, view it on GitHub https://github.com/ben-denham/labtech/issues/26, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAN25UYJZNGVUPOU2YJXFE3ZOV54PAVCNFSM6AAAAABLTGRMA6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGQZTIMJWGMYDEMI . You are receiving this because you are subscribed to this thread.Message ID: @.***>