Batch Jobs

[bcwaldon]

Explore how to handle batch jobs (units with Type=oneshot)

[bcwaldon]

Probably encompasses https://github.com/coreos/fleet/issues/240

[jonboulle]

I'm considering #308 a blocker for this, as properly implementing this feature turns out to require a lot of changes to the CLI plumbing.

[bcwaldon]

@jonboulle fair enough

[yichengq]

What syntax may be used for batch jobs? I am still a little confused about the CLI blocking. :P

[jonboulle]

Are you talking about CLI syntax or configuration file syntax?

[yichengq]

@jonboulle CLI syntax. I think it is great if we could have a consensus on what CLI should support for the feature.

[sbward]

As a user, I expected fleetctl start my_oneshot.service (with Type=oneshot) to launch the unit on the cluster every time I ran the command, without needing to run fleetctl stop my_oneshot.service before launching it again. It would be great to be able to run the same oneshot service N times in a row, asynchronously launching them into the cluster. Although this makes me wonder how I would get the journal entries from each of my oneshots...

[termie]

yay! glad this is already being talked about a bit, i guess the kinds of jobs I am starting would be called a batch job, which is not clearly defined here but i'll offer my description and maybe we can update the issue.

I have a job that runs once and then exits and, when complete, cleans up after itself (removes itself from fleetctl list-units)

[termie]

(I suppose that is basically the definition of "oneshot" but the issue is still somewhat terse)

[xied75]

Should this be merged with #562?

[Syerram]

Any updates on this? oneshot feature is critical for our systems. @termie somewhat gave a high level description of batch jobs, but I elaborated a bit more below

Existing Arch (without CoreOS)

• user submits a request to run code
• code wrapped and additional meta data sent via RabbitMQ
• a node in the cluster receives the messages and accepts (assuming it has the capacity) and launches the docker container with the wrapped code/meta data.
• docker executes and dumps logs in the mounted folder, pings back to the daemon on that node as complete.
• daemon returns back the data [return URL was in the meta-data)

With CoreOS

• user submits a request to run code
• code wrapped with meta data dumped in etcd with unique key
• launch the service via fleetctl with a unit launching docker container and passing the unique key (and type set to oneshot)
• docker services retrieves metadata from etcd, executes and sends results to the master
• docker service exits and the service is scrapped and removed from the list-units.

I think we are missing the last part i.e. the service once finished is completely removed from fleet. Next invocation will start the service as a new unit.