Open onnozweers opened 2 months ago
I see now that other schedulers have already been anticipated: https://github.com/dCache/dcache/blob/b772cd54a1ae45453da86dd87649add89f6e4745/skel/share/defaults/bulk.properties#L50C1-L58C92
# ---- Algorithm for promoting requests to active.
#
# Implementations:
#
# - org.dcache.services.bulk.manager.scheduler.LeastRecentFirstScheduler:
# selects the least recent requests to run in order of arrival
# (first-come first-served).
#
bulk.request-scheduler=org.dcache.services.bulk.manager.scheduler.LeastRecentFirstScheduler
Dear dCache devs,
My impression is, that currently, the bulk service processes requests in the order they come in. The number of threads (concurrently active requests) is limited, so it can happen that a few stage requests are keeping all threads busy, and release (=unpin) requests are not getting through because they remain queued.
So, you might have a situation where, for instance, Atlas is staging a lot, and the stage pools fills up, and the unpin requests are waiting for the stage operations to complete, so they are not getting through, although they would be the solution for the space congestion.
I've had to do some tuning to make this work better:
With 1000 threads, I saw that unpin requests could get through so they could free up space so that staging could continue.
So I'd like to suggest that release requests have priority over stage requests. Release requests take very little time and they free up the necessary space for staging to succeed. If that is too difficult or undesired, it might help a lot to increase the default number of threads. And perhaps while we're at it,
bulk.limits.max-requests-per-user
could have a bigger default too (like 50000 instead of 5000). Atlas would be very happy.An alternative idea might be to reserve some threads for only
release
requests, so they never stall when stage requests are stuck.Cheers, Onno