m3dev / gokart

Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline.
https://gokart.readthedocs.io/en/latest/
MIT License
318 stars 57 forks source link

Add cache existence check to wrap_with_run_lock #331

Closed mski-iksm closed 1 year ago

mski-iksm commented 1 year ago

Changed feature

I've added cache existence check to wrap_with_run_lock

What is wrap_with_run_lock?

wrap_with_run_lock is a wrapper function for TaskOnKart.run(). This is used to wrap TaskOnKart.run(), when you want to use efficient task cache collision lock https://gokart.readthedocs.io/en/latest/using_task_cache_collision_lock.html#advanced-using-efficient-task-cache-collision-lock

What is the problem of original code?

Suppose you are to run same task on different nodes at the same time.

In this case, node1 and node2 will do the same thing with the same result, which is inefficient.

By using efficient task cache collision lock, TaskA.run() will not run at the same time.

However, in the original code, after node1 finished running TaskA, node2 will run TaskA again. Cache collision will not happen, but this is still inefficient, because node2 didn't need to run TaskA again.

How to overcome this problem?

I've added cache existence check just before TaskOnKart.run() to prevent running run() again. When the cache file is found at the runtime, run() will be skipped.

mski-iksm commented 1 year ago

I will close this PR, because cache existence check must be separated from cache collision lock wrapper.