nils-braun / b2luigi

Task scheduling and batch running for basf2 jobs made simple
GNU General Public License v3.0
17 stars 11 forks source link

Job status cache for gbasf2 #174

Open meliache opened 1 year ago

meliache commented 1 year ago

The get_job_status call in the Gbasf2Process is rather costly and the gbasf2/dirac servers can only handle limited requests at a time. Therefore it would be good to create a job status cache by implementing a sub-class of BatchJobStatusCache, similar as to what has been done for htcondor and LSF.

It's possible to check the status of multiple projects with a singlegb2_job_status call and I'm sure we can do this with out custom script as well. We can then fill the cache will the statuses of all jobs in all user projects. If we inherit from BatchJobStatusCache, then by default the cache gets re-filled every 20 seconds (the time-to-live, TTL), but for the grid I would suggest to set a longer TTL.