Closed harrypuuter closed 1 year ago
Hey @harrypuuter ,
thanks for opening this issue! The caching was added at some point exactly for cases like that, but it is obviously not doing what it should right now.
I think this can be fixed, but my time is a bit limited the next couple days. I'm trying to fix this asap and get back to you for a final confirmation of the fix :)
Hi @riga - Thanks a lot for looking into this. If I can be of any help, testing stuff etc. let me know !
Finally found the time ...
And indeed it was a caching "feature". So far, the cache was not forwarded from the workflow to its branch tasks (and also not among branch tasks themselves) so that each of them tried to build up its own branch map, not knowing that the associated workflow already did that.
I added this forwarding and it's way faster now as expected. Could you also try with the feature/forward_cached_workflow branch?
On a different note (and sorry if this was just for the sake of creating an example), is there a reason why you
collection._flat_target_list
?Closing and merging for now, but feel free to still report if this fixes your issue.
Hey, sorry for the late reply. With the fix, everything now works as intended, thank you very much !
Hi,
I am currently facing a problem setting up a more complex workflow task, where I observe very bad scaling. I have created a minimal example that represents the actual workflow I am trying to implement. I am using the current master branch of law.
I have a WrapperTask, which requires WorkflowZero, which requires WorkflowOne, which requires WorkflowTwo.
Workflow Two is used to process a number of files (e.g. 100, 500, 1000). Workflow One is used to process the output of workflow Two (one task per file) Workflow Zero is used to process the output of workflow One (one task per file)
In all workflows, the
NestedSiblingFileCollection
is used. In order to create the branch map for workflow one and zero, the input from the processing workflow is used. The example can be found below.The problem with this setup is that it gets unusable when using a larger number of files. The print-status command is scaling exponentially with the number of files in workflow two:
100 files : ~0.4 s 500 files: ~15 s 1000 files: ~95s
Example:
From what I found in the law implementation, this problem seems to be related to the fact that the
branch_map
is not cached properly, and in this nested setup, many calls ofcreate_branch_map
are made, slowing down the whole determination. In the example, I use local file targets; when using wlcg targets, the problem gets even worse since the existence checks are much more expensive.I tried to identify the underlying issue with the
branch_map
caching and it seems like this https://github.com/riga/law/blob/fd750b1df3cd41df1d0d08d945ef8182ac39d7b1/law/workflow/base.py#L623 caching does not work after changing a task toas_workflow()
and as as result, thebranch_map
has to be recreated by calling thecreate_branch_map
function of each workflow. Since we have three nested workflows here, where the inputs are used to create each branch map, the problem is further amplified.Is there another way, how such a workflow should be setup ? Any input is appreciated !