Background Section

[SteVwonder]

We need the "elevator pitch" for Flux in our docs.

What is Flux? What problems is it trying to solve? Why is it better than the "competition"?

We have lots of research papers that we can probably copy/paste from.

[SteVwonder]

One recent email thread contains a potential answer to the following question:

What is Flux? What problems is it trying to solve?

Flux is a next-generation scheduler and resource manager designed to tackle four major challenges: job throughput, co-scheduling, job coordination/communication, and portability. Flux’s ability to nest within itself and the allocations of other resource managers means that it is highly portable (can run under any RM, like Slurm and LSF) and that it can achieve high throughput by spinning up lots of nested instances to spread the scheduling load across. And each of those nested instance runs a full featured scheduler (not just a simple parallel launcher), meaning users can co-schedule arbitrarily complex jobs at any level within a hierarchy of Flux instances, solving the co-scheduling challenge. Finally, Flux has APIs for everything (not just CLI) which helps with the job coordination/communication and portability challenges. For more details, we recommend taking a look at our recent journal paper which highlights how workflows at LLNL are using Flux to solve several of their major challenges: http://flux-framework.org/papers/Flux-FGCS-2020.pdf.

[dongahn]

Sounds good.

We have lots of research papers that we can probably copy/paste from.

I think being succinct would be ideal as our background materials even using bullet formats. From my perspective,

• First class support for modern scientific workflows with hierarchical nesting, user-level instances and rich API set, and portability is huge (Flux uniquely solves this)
• Graph-based approach to schedule diverse set of resources easily while allowing for scheduling specialization have also proven to be effective

We have developed other system-level solutions and we might want to think about a few differentiator as system level solutions as well and add them to the background.