Scheduling/Framework session schedule

[betatim]

This issue is to organise the schedule of the Scheduling/Framework sessions.

Please post your contributions/ideas below and we will update the table

Monday	Tuesday	Wednesday	Thursday
joint session with "event model", collect ideas for short hacks to be worked on Tuesday and Thursday.	hacking	Standup and talks	hacking

[ishapoval]

Contribution: "Low latency, scalable concurrency control in Gaudi Hive"

The move from sequential to concurrent (multithreaded, task-based) Gaudi complicates the problem of decision making in the space of control flow and data flow constraints of algorithm execution precedence and requires a dedicated concurrency control system for it. Concurrent environment imposes the requirements of at least 2D scalability and soft real-time reasoning on this system. In this contribution I will describe a new, low latency and scalable, concurrency control system (CCS), deployed by me to the Gaudi Hive prototype. I will present its new algorithmic approach, performance measurements, as well as will point out its advantages over the previously used CCS.

[ishapoval]

Contribution: "Predictive scheduling for peak throughput in Gaudi Hive"

The Gaudi Hive prototype introduces two dimensions of concurrency: at intra- and inter-event levels. Previous generation of the prototype was based on "myopic" concurrency control with only reactive decision making effectively possible at the intra-event level. Due to significant heterogeneity/asymmetry of typical LHCb/ATLAS algorithms' precedence constraints and some degree of precedence freedom for a given data processing workflow, this may lead to frequent degradation of intra-event disclosure dynamics. In such situations, the only strategy to saturate the framework's occupancy is to go for higher degrees of inter-event concurrency, which, in turn, leads to increase of framework's overall overhead and, subsequently, to degradation of its peak throughput. In this contribution I will present an alternative scheduling strategy, which utilizes the ability of the new generation concurrency control for efficient proactive decision making to generically maximize the intra-event concurrency disclosure dynamics by predicting optimal intra-event execution plans. This allows to minimize the need in excessive degrees of inter-event concurrency, and thus to maximize the peak throughput of the framework.

[betatim]

We could have two talks of about 15 each on Wednesday morning about this.

Some thoughts:

• How much of this is still applicable if we switch to a task based system?
• You should write a plain english version of those two summaries to maximise the number of people who read it and then think "oh that is interesting" so they come and listen.
• Could you explain in plain english what you mean with a "concurrency control system"? Is it a scheduler?

[ishapoval]

Given that the subject of the contributions just cannot be any closer to the topic of the session and is just it, may I have 20m/talk? I could well fit in 15m, but I prefer some spare time not to rush. More over, the sessions on Wed and Thu seem to be completely empty anyway for the moment..

Correspondingly to bullets:

• It is all applicable and meant purely for this. As I said in the 1st sentence of the 1st summary. Gaudi Hive was task-based from the very beginning.
• I think I kept it "plain" as much as I could, especially in the 1st summary. Further simplifications would have led to some loss in what my message is. My purpose was to describe concretely what the contributions are about, and I used well-established domain terminology for that. I think this is the right style for keeping ourselves in sync with this domain, existing out there for decades already, we should not put ourselves in our own terminological cocoon. I think for those who are really interested in this topic the titles are quite telling: from them it is clear that it is about Gaudi throughput, one of the main, if not the main, characteristic of any framework of such kind.
• Concurrency control is concurrency control. Hierarchically, it is part of scheduling, which has a wider sense. The aim of concurrency control is to answer the question of which tasks can be scheduled with no conflicts in conditions of tasks being not independent, like in Gaudi, where we have control flow and data flow task dependencies.