If a task has a lot of subtasks, each demanding for a lot of computation, it is very likely that the processor will not be able to finish the entire task (essentially all the subtasks) due to limitations in memory or computation resources. So, the task will break at some point in the middle. For example, we can have in total 100 subtasks and after a 'doit' command only 30 of them have been performed successfully, and the task has been killed in the meantime of executing the 31-th subtask. In this situation, because in pydoit the update of any task will only happen after the entire task (all of its subtasks) has ended successfully, obviously the task won't be updated. This means, another doit command restart to process all the sub-tasks from the begining again, and the 30 subtasks that we did finish in the previous doit round are lost (not considered to be updated). And the story repeat itself, because any time we run doit command, only few of the subtasks are done successfully and the whole task breaks at some point.
So, it would be very useful in such a situation to tell the doit that, it makes an update of the doit.dat file after each subtask is successfully done. Or even better would be to say after each N subtasks the update of doit.dat file takes place, in order to save time just in case that the update will take a lot of time, especially if the number of subtasks is too large.
As far as I had a look into the documentation of pydoit, this feature does not currently exist in Pydoit.
Upvote & Fund
We're using Polar.sh so you can upvote and help fund this issue.
We receive the funding once the issue is completed & confirmed by you.
Thank you in advance for helping prioritize & fund our backlog.
If a task has a lot of subtasks, each demanding for a lot of computation, it is very likely that the processor will not be able to finish the entire task (essentially all the subtasks) due to limitations in memory or computation resources. So, the task will break at some point in the middle. For example, we can have in total 100 subtasks and after a 'doit' command only 30 of them have been performed successfully, and the task has been killed in the meantime of executing the 31-th subtask. In this situation, because in pydoit the update of any task will only happen after the entire task (all of its subtasks) has ended successfully, obviously the task won't be updated. This means, another doit command restart to process all the sub-tasks from the begining again, and the 30 subtasks that we did finish in the previous doit round are lost (not considered to be updated). And the story repeat itself, because any time we run doit command, only few of the subtasks are done successfully and the whole task breaks at some point.
So, it would be very useful in such a situation to tell the doit that, it makes an update of the doit.dat file after each subtask is successfully done. Or even better would be to say after each N subtasks the update of doit.dat file takes place, in order to save time just in case that the update will take a lot of time, especially if the number of subtasks is too large.
As far as I had a look into the documentation of pydoit, this feature does not currently exist in Pydoit.
Upvote & Fund