There is some time between the moment a node is processed and the moment its
children are fetched.
Assuming that several nodes with a similar set of children are processed
during this time, this means that the same children may be fetched and queued
for further processing more than once (with ProcessNode).
It would seem that in a context of dags with many branches that are syncing,
many interleaved, this might result in the reprocessing of multiple nodes, all
of them with many children, causing general stress in the system (particularly
making the fetching of children slower, and thus exarcebating the problem by
extending the time between processing and fetching).
This attempts to mitigate this by skipping children which have been claimed to
processing by some other job.
There is some time between the moment a node is processed and the moment its children are fetched.
Assuming that several nodes with a similar set of children are processed during this time, this means that the same children may be fetched and queued for further processing more than once (with ProcessNode).
It would seem that in a context of dags with many branches that are syncing, many interleaved, this might result in the reprocessing of multiple nodes, all of them with many children, causing general stress in the system (particularly making the fetching of children slower, and thus exarcebating the problem by extending the time between processing and fetching).
This attempts to mitigate this by skipping children which have been claimed to processing by some other job.