search

Netflix / genie

Distributed Big Data Orchestration Service
https://netflix.github.io/genie
Apache License 2.0
1.7k stars 365 forks source link

Initial support for expanding configurable execution environments for jobs #1171

Closed tgianos closed 2 years ago

tgianos commented 2 years ago

Expose ability for defaults, command defaults and job request overrides for compute resources:

Also create new mechanism that allows admins and users to define sets of container images that should be resolved into a final payload to send to the agent launcher. These images can then be refined (into a Kubernetes pod for example) before launching the final job. TitusAgentLauncherImpl was modified to currently only take the main image but will be extended via adapter internally to launch as something similar to a pod.

Images can have:

Priority order of resolution: job request -> selected command -> system defaults

coveralls commented 2 years ago

Coverage Status

Coverage increased (+0.1%) to 93.916% when pulling 366525f43952ec11214fcc30ea8fd5793719eab5 on tgianos:sparkOnTitus into b2af438ec04ee1e41697dcf2239af1281d7fc4ae on Netflix:master.