JuliaParallel / Dagger.jl

A framework for out-of-core and parallel execution
Other
621 stars 67 forks source link

Sch: Temporarily disable chunk cache #465

Closed jpsamaroo closed 7 months ago

jpsamaroo commented 7 months ago

The scheduler chunk cache has been around for a while, but unfortunately was never really correct in the face of mutating data. If data is mutated by any Dagger task, Dagger's chunk cache may still serve the stale, non-mutated data copy. This issue has come up in a number of places, and so it needs to be resolved now before causing further issues. A proper fix is still in the works (as part of a larger set of changes), but in the meantime, this hotfix will have to do.

jpsamaroo commented 7 months ago

Aside: according to Dagger's current semantics (that tasks are functional in nature, and so arguments should not be mutated), this is technically correct behavior, but it is also confusing and hard to disable when you do need mutation. It's also correct to not cache chunk values, so it's valid to remove it. Future changes will enable mutation of arguments with some simple argument annotations, so this work is really preparing for that, and avoiding a quiet, hard-to-detect footgun until that support lands.