ORNL / DataFed

A Federated Scientific Data Management System
https://ornl.github.io/DataFed/
Other
18 stars 14 forks source link

Task block edge proliferation #786

Open dvstans opened 3 years ago

dvstans commented 3 years ago

During a period of very heavy task loading, there were roughly 50,000 tasks queued for many duplicate operations (transfers, some deletes). At this time there were ~250,000 task block edges. This is a 5:1 ratio which seems too high. Need to evaluate the block code and make sure it is not generating spurious blocks and determine if there is a more efficient way to implement blocking that does not create so many edges.