Portable / pluggable job submission system

remram44 commented 5 years ago

Project description

Job submission systems are common in scientific computing (e.g. TORQUE or PBS). They allow you to start a job (some script) and then later on check the status of that job and get the output and log for it.

However this can be useful for more than just big companies and clusters. It is common that you might need to download and transcode a few videos, or render some 3D scenes. It would be useful to have some command-line tool that can run some script on a remote machine (via SSH, or on a cloud like AWS or GCP) and get the results.

What I have in mind is a command-line tool that can support multiple backends (AWS, GCP, random machine via SSH, legacy job-submission systems via QSUB, ...), similar to rclone but for computation.

Advanced job-submission features such as queuing and resource allocation etc are probably not needed, they can be delegated to the underlying system.

Existing work

I actually made a prototype for that long ago (tej, for Tiny Extensible Job-submission) in Python but it only supports SSH. Today I find myself wanting to run tasks on the cloud and I'm wondering if something can do that, or if it's worth making, hence this issue.

There also exist tools for provisioning, such as Terraform or Chef/Ansible, but those only set-up infrastructure and have no support for checking a job's status or getting results.

Relevant Technology

Most anything could be used, but I think Go or Rust are good solutions since they can produce static binaries that can be run on remote machines to monitor jobs etc. But a scripting language would also work if it's available on the remote. tej uses shell scripts only, but that's very limited.

Complexity and required time

Complexity

[ ] Beginner - This project requires no or little prior knowledge of the technolog(y|ies) specified to contribute to the project
[x] Intermediate - The user should have some prior knowledge of the technolog(y|ies) to the point where they know how to use it, but not necessarily all the nooks and crannies of the technology
[ ] Advanced - The project requires the user to have a good understanding of all components of the project to contribute

Required time (ETA)

[x] Little work - A couple of days
[ ] Medium work - A week or two
[ ] Much work - The project will take more than a couple of weeks and serious planning is required

open-source-ideas / ideas