search

flux-framework / flux-sched

Fluxion Graph-based Scheduler
GNU Lesser General Public License v3.0
89 stars 41 forks source link

batch consume messages, immediately process cancellation #1226

Closed trws closed 5 months ago

trws commented 5 months ago

This is in progress, but so far takes the 10,000 allocs and 10,000 cancels from some terrifyingly long amount of time to 36 and a half seconds. It's using a combination of @garlick's batching suggestion and immediately processing cancels by interrupting the scheduling loop if necessary. Pauses may be too long, probably needs tuning, but clearly better. Popping it up here before we hop on the plane back from sweden this morning, so I'll be in and out of communication today.

trws commented 5 months ago

This design turned out to be a bit of a dead end for match performance reasons, new PR coming that makes all of our backfill queues asynchronous.