ORNL / DataFed

A Federated Scientific Data Management System
https://ornl.github.io/DataFed/
Other
18 stars 14 forks source link

System - Must limit number of tasks being processed by core service #432

Open dvstans opened 4 years ago

dvstans commented 4 years ago

The core task manager currently loads all tasks that are ready to run into memory (basically an string ID + 32 bytes of control info). It's possible for there to be millions of tasks ready to run; thus a limit/high-water mark is needed to prevent using all memory. When this limit is reached, new tasks would be written to the database, but not loaded until a low-water mark of running tasks is reached.