search

2i2c-org / features

Temporary location for feature requests sent to 2i2c
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Batch job support for Pangeo hubs #15

Open rabernat opened 1 year ago

rabernat commented 1 year ago

Context

Pangeo hub users often want to put a long-running job into the background. Instead, on our hubs today, they have to essentially keep a notebook open all the time for these long running jobs. This leads to awkward and inefficient workflows, such as postdocs not being able to close their laptops for days.

cc @paigem, @jbusecke

Proposal

I propose we install kbatch on the Pangeo hubs. We discussed this idea quite a while back, but I can't find any record of that conversation.

Updates and actions

No response

jbusecke commented 1 year ago

This would be an awesome addition to the hub for pretty mich every project I am involved. Enthusiastic 👍, and happy to test!

jmunroe commented 1 year ago

I agree that batch submission is an important class of computing for many problems. And it does appear that kbatch would be one way of providing that functionality.

But, from what I am reading, kbatch is a relatively simple wrapper for submitting a job to a kubernetes cluster that lacks features compared to something like slurm or prefect. I think it makes sense to use for automatic running of a notebook in a lights-out way, but is its intended use case only for relatively short jobs? Is kbatch still appropriate for running multi-day batch computing?

While it is of course no worries to experiment with kbatch if that solves people's immediate problems, are there other scheduling systems that we should be considered? I am worrying about things like setting limits on long-running jobs, resources allocation between users, reporting and logging. I think other hubs also are discussing "batch" computing as well so I'll going to add this a feature request on our weekly 2i2c Product and Engineering meeting to see if there are other options that should be considered.

rabernat commented 1 year ago

All good questions @jmunroe. @yuvipanda worked quite a bit on kbatch and decided it was the sweet spot in terms of complexity. But happy to align whatever tools 2i2c wants to support here.