TaskOnKart.complete_check_at_run is a parameter to control whether to check completed cache file existence at TaskOnKArt.run().
If complete_check_at_run==True, when the worker attempt to run the task, it will check the cache file existence, and if exists, run() will be skipped.
Why is this needed?
In the original code, complete check is only evaluated at task scheduling, which is at the very beginning.
However, cache status may change when it comes to run().
This often happens when jobs in multiple nodes runs the same task.
When node1 and node2 runs the same task, the first finished node will create the completed cache, but the other node may rerun the task and recreate the same cache file.
It is okay to run same task again, but it is not efficient.
To prevent running the completed task again, I've implemented TaskOnKart.complete_check_at_run option.
Hi-king
commented
10 months ago
What is this?
TaskOnKart.complete_check_at_run is a parameter to control whether to check completed cache file existence at TaskOnKArt.run(). If complete_check_at_run==True, when the worker attempt to run the task, it will check the cache file existence, and if exists, run() will be skipped.
Why is this needed?
In the original code, complete check is only evaluated at task scheduling, which is at the very beginning. However, cache status may change when it comes to run().
This often happens when jobs in multiple nodes runs the same task. When node1 and node2 runs the same task, the first finished node will create the completed cache, but the other node may rerun the task and recreate the same cache file.
It is okay to run same task again, but it is not efficient. To prevent running the completed task again, I've implemented TaskOnKart.complete_check_at_run option.