Jobson 1.0 Feature Roadmap

[adamkewley]

The aim of 1.0 is to deliver a stable system that covers most use-cases.

• I would like to deliver 1.0 by the end of 2018
• However, most of the features below are not necessary for jobson's internal requirements, which means development of these features will be done out-of-hours (that is, slowly)

jobson-1.0.0 roadmap:

• OAuth & stateless JWT authentication support:

  • Enables Jobson to be integrated into larger system/institutes in a standard way
  • OAuth would make it easier to integrate Jobson in larger institutes and teams
  • Stateless JWTs would make it easier to integrate Jobson as an internal microservice inside a trusted network

• Authorization:

  • Enables Jobson to restrict API access/responses based on the user's permissions
  • The system should support groups and group permissions
  • Implemented in terms of "read" / "write" (abort, etc.) permissions

• System-wide execution hooks:

  • Used to extend Jobson in a language-independent manner
  • Similar idea to git hooks: separate executables that are ran at various stages of a jobson job
  • Some hooks (e.g. before-running-job) would prevent a job from running if the exit code from the executable is nonzero
  • Examples: A python script that tweets whenever a job completes. A ruby script that validates job names to ensure that all jobs submitted system-wide conform to organizational requirements. etc.

• Stateless HTTP and Websocket API:

  • Makes Jobson more resilient to server resets and failures and facilitates putting it behind a load balancer.
  • The server implementation needs to be refactored to get all state+events from a dedicated layer (a filesystem, message queue, etc.)
  • The HTTP API is already mostly stateless: the state is usually loaded from the filesystem (which multiple servers could share); however, job execution is stateful (a particular server does the executing).
  • The Websocket API is currently stateful: it directly receives events from the JobManager as job execution occurs. Another server in the cluster cannot view these events. Therefore, the Websocket API will break when put behind a load balancer unless "sticky" load balancing is done. This pushes complexity up to the sysadmins/devops guys, which is non-ideal.

• Job system tests:

  • Makes devops and sysadmin tasks easier
  • Spec developers will be able to provide a tests/ or tests.yml test suite inside their job folder
  • Those file(s) will contain test inputs for the job and a set of expectations (e.g. "the job fails", "the job takes under 10 min", "the job produces a .tar file with a size greater than 10 MiB")
  • A jobson test CLI command will be implemented to run the test suites, returning a non-zero exit code if a suite fails

• Unified output API:

  • Makes developing downstream API clients easier
  • Currently, stdout and stderr are treated as special cases of a job output and have their own API entry points (e.g. /v1/jobs/{job-id}/stderr)
  • In reality, stdout/stderr are just job outputs that happen to be collected via pipes. They should have all the features available to standard job outputs and, vice versa, standard job outputs should have all the features available to stdout/stderr (e.g. a Websocket API).
  • The job output API should be designed to handle not-yet-persisted outputs, partial outputs, etc. (as is already done for stderr/stdout)

• Remove language-dependant features:

  • Remove any outward-facing Java APIs (e.g. custom authentication), providing a language-independent alternative (e.g. OAuth)

• HTTP API Improvements:

  • All entry points should have relevant HATEOAS hrefs
  • timestamps should be removed from job details and placed into a separate API entrypoint (e.g. /v1/jobs/{job-id}/timestamps
  • A job details response from /v1/jobs and /v1/jobs/{job-id} should contain status (currently: downstream devs trawl timestamps, which is a PITA)

• Documentation:

  • HTTP API documentation available via the API itself
  • CLI documentation available online and via the CLI itself
  • Additional documentation (pages, examples, FAQs) should be implemented via a site generator (not reliant on github)

jobson-2.0.0 wishlist:

The ideas I have for 2.0 are going to be much more complex to implement. They require that 1.0 is:

• Stable: so 1.x maintenance programming doesn't eat up limited dev time
• Simple: so technical debt doesn't make 2.0's features too complex to implement

Features

• User-defined input data types:

  • Allows the user to specify custom input data types
  • Would require a language definition, parser, validator, etc.
  • Unclear how runtime validation would be implemented in a language-independent manner

• Job distribution:

  • Allows jobs to be distributed, with jobson handling node failures, attempts, scheduling, etc
  • Would require a clear distribution definition that matches Jobson's execution model (forking). A network and communication protocol would need to be defined, alongside other concerns
  • Would be a strong feature but bumped to 2.0 because core system needs to be rock-solid before putting the engineering effort into this

• Job composition:

  • Allows jobs to be composed from other jobs, in a kind of "pipeline" style, with jobson handling input/output forwarding, distribution, IDing etc. in a way that's transparent to API clients
  • Would also be a strong feature but bumped to 2.0 because, again, core system needs to be rock-solid before putting engineering effort into this

• Resource management:

  • Makes jobson schedule jobs based on resource needs (nodes, cores, memory), rather than using a fair queue
  • Would be useful to HPC devs because they develop a wide range of HPC apps that use differing amounts of GPUs, CPUs, etc. and jobson's queue is not sensitive to that
  • Bumped to 2.0 because resource management and scheduling can get very complicated and, again, the rest of the system needs to be solid before doing this

[adamkewley]

Addition to the v2 bucket list:

• Job attempts: each job (identified by job-id) has 1-n attempt IDs, which allows a single job to have multiple attempts. Some prod deployments with jobson1 use flakey connections, clusters, etc. which are failure prone. However, this can't be patched into jobson1 without either hacking the stdout/stderr outputs or breaking the API (to support attempts)

[adamkewley]

This roadmap has been dropped because it's now actual 1.0.0 crunch time, so I can actually write what 1.0.0 will have, rather than what I'd like it to have in 12 months time. Closed because this is stale.