Provide for dependencies between tasks in a group

[mfischer-zd]

Tasks in a group sometimes need to be ordered to start up correctly.

For example, to support the Ambassador pattern, proxy containers (P[n]) used for outbound request routing by a dependent application may be started only after the dependent application (A) is started. This is because Docker needs to know the name of A to configure shared-container networking when launching P[n].

In the first approximation of the solution, ordering can be simple, e.g., by having the task list in a group be an array.

[ChrisHines]

+1

Batch jobs need task dependencies to manage data-flow between tasks.

I would expand this to allow dependencies between tasks in a job rather than just a task group. Data flow is not constrained to a single node.

[suyash-repo]

Any tentative date to add this enhancement feature in Nomad?

[dadgar]

@suyash1983: we don't have any dates yet but it will most likely be something that comes in 0.4

[blalor]

This is really going to be core to a number of use cases, like allowing EBS volumes to be mounted on a host before the application starts.

[c4milo]

or pulling down certificates and secrets from Vault before starting a given service.

[vrenjith]

+1 This will be a great addition for large enterprise cluster deployments

[steve-jansen]

+1

Running consul-template alongside an app...

[dadgar]

@steve-jansen You can run consul-template currently. We do this in our Nomad deployment. Essentially you have a script that runs consul-template which produces your config, and then runs your binary

[steve-jansen]

@dadgar nice, thanks for sharing. It's always great to hear how a company uses their own products. I imagine consul-template creates the env vars in the job config...

One catch for us, we aspire to have consul-template rewrite the config in response to changes in a Consul k/v, or more importantly, when Vault rotates a secret. We're trying to have consul-template signal the co-scheduled task to reload its config.

Indeed, my need for task dependencies is very narrow. In a perfect world for me, Nomad would have integration with Consul K/V and/or Vault. My need is config that updates dynamically for tasks. That would eliminate my need for task dependencies.

[dadgar]

@steve-jansen consul-template can supervise the process and restart/signal it whenever the config changes. @sethvargo to verify

[sethvargo]

Hi @steve-jansen

Vault currently does not support blocking queries, which is what Consul (and CT) use to give you that "real-time" trigger when something changes. CT will renew secrets at lease_duration/2.0, but it's not currently possible to trigger a change in Vault and have that immediately notify another process. There is an open issue in Vault for blocking queries. Please note this was an intentional design decision in Vault for performance reasons, and not a bug in CT or Vault.

CT will start the process, but it is not a supervisor or monitor. CT has the expectation that the given command will return within 30s, typically by delegating to some supervisor. If you're running in pure Docker, CT can optionally cleanup PIDs and act like PID 1, but it doesn't monitor the process itself (which is best monitored by the scheduler anyway).

[OferE]

+1 - any updates on this?

[nugend]

I'd also like to see this land. I'm evaluating nomad for service orchestration and it would be very nice to be able to explicitly express dependencies.

Alternatively, or as a stop gap, maybe a recipe for doing this with the existing APIs could be written up (#1065 mentions this is possible)

[mbravorus]

Currently, the use cases which require dependencies and cron-like control tend to push me towards airbnb's Chronos which implies Mesos. I would be delighted if I could just use Nomad.

Recurring jobs are possible (starting from #540, ultimately described in https://www.nomadproject.io/docs/job-specification/periodic.html), but in a very common scenario where periodic tasks need interlocking dependency control, there is no way to do it with Nomad currently (or I didn't manage to find it)

[nugend]

Given that this doesn't seem to be a high priority, could someone from the project briefly explain the suggestion about how to achieve this with existing APIs as mentioned in #1065?

[dadgar]

@nugend You would essentially need to put a wrapper around the task that uses the allocation endpoint to determine if the task it should wait for has finished successfully and then start itself. It is non-trivial amount of work though

[RobertAtomic]

+1 for DAG support... It's quite important for my use case.

[tduffield]

I too would love to see this.

[SunSparc]

+1 for task sequence/dependency

[nugend]

For anyone really pining for this, there is a slightly easier method provided you are registering the upstream tasks as services: use a template stanza to create a run script and make that the command of your task. Then you can use a conditional dependent on the service being available to either launch the actual process or sleep forever.

Works pretty well, though it does require a bit of scripting (and I’m not sure how to do it in Windows)

[vrenjith]

Or use containerpilot if you are using docker driver of nomad. https://github.com/joyent/containerpilot

On Mon, Oct 23, 2017 at 6:32 PM, Daniel Nugent notifications@github.com wrote:

For anyone really pining for this, there is a slightly easier method provided you are registering the upstream tasks as services: use a template stanza to create a run script and make that the command of your task. Then you can use a conditional dependent on the service being available to either launch the actual process or sleep forever.

Works pretty well, though it does require a bit of scripting (and I’m not sure how to do it in Windows)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hashicorp/nomad/issues/419#issuecomment-338651471, or mute the thread https://github.com/notifications/unsubscribe-auth/AB_wuV8EgzlMZFkFFd_XrQ_pplUPndUrks5svI5kgaJpZM4Gie-j .

-- Regards, Renjith

[abohne]

Is there any update on where this might fall on the roadmap?

[dadgar]

Likely in 0.9

[cl0udgeek]

come on 0.9!

[Miserlou]

How likely is likely? :)

[prandelicious]

Any news on this?

[dadgar]

As mentioned early this is either in 0.9.x or 0.10.x.

[ershad]

Would love to have this feature!

[lmayorga1980]

:+1: for 0.9

[gopi-ideas2it]

Waiting for 0.9 in 2019 @dadgar. Any update on this would be highly appreciated.

[c4milo]

People wanting to use Nomad as a workflow orchestrator should look into using Apache Airflow instead, scheduling tasks into Nomad. There is a Nomad Operator in the works here: https://github.com/apache/airflow/pull/2708.

[rgevaert]

Is 0.9.x still targeted?

[jippi]

Haven't seen it in 0.9 changelog, so probably 0.10 :)

[johnzhanghua]

+1

[vadfabi]

waiting on this vital feature ...

[cgbaker]

Task Dependencies is currently on the roadmap for 0.10

[djenriquez]

+1

[dpisklov]

@cgbaker hi Chris, would you be able to provide a (rough) date for 0.10?

[cgbaker]

@dpisklov , we are still trying to figure out which features at going into the first 0.10.0 release and which into later point releases. feel free to keep checking back here!

[shantanugadgil]

Please consider prioritizing this for a more recent release than 0.10.0 :trophy:

Task dependency is something that many folks (including me) could/would benefit from in designing application deployment workflows without getting into hacks like writing sentinel files for subsequent tasks.

Thanks and Regards, Shantanu Gadgil

[obikay200]

Please can this be prioritised asap as we are trying to convince clients to move over to nomad and away from jenkins pipelines. they are not building but executing tasks! so a sequential ability would improve things 10 fold.

[kainoaseto]

This would be really great to have to solve deployment issues where there are implicit dependencies between tasks, such as container networks, that nomad will not prioritize for. Along with all the other use-cases mentioned here, this would be a very welcomed addition to Nomad.

If this can be prioritized even ahead of 0.10.0 that would be fantastic but at least making the 0.10.0 release would be extremely welcomed!

[manish-panwar]

Are we ever going to implement this feature - 4 years is long time. I think this is an important feature. We are evaluating Nomad as our enterprise orchestrator - but now I am thinking if we are making the right decision to use Nomad.

[kcwong-verseon]

This is not an easy matter. I'd rather the HashiCorp folks take the time to think this through. Since Nomad has job types, one can see task dependency having different implications for different job types. For service jobs, the simplest approach would forego task dependency and just let restarts sort things out. However, in the use-case as stated in this issue, things like namespace dependence cannot be addressed via restart. Other use cases that require dependency management include setup and teardown for a task, which doesn't really work well with having a job-type of service (a more viable solution may be pre- and post- task "tasklets".)

In the case of batch jobs, a task group may represent a pipeline in which a subsequent task may depend on the output of a previous one.

Add to these complexities is the handling of various failure scenarios. What to do if a task in a group fail? What to do if you need to drain the node? What about Nomad agent crashing? Thank goodness I don't have to noodle on how to deal with them in a comprehensive manner.

One thing I really like about Nomad is a reasonable progression in feature addition (I'm staring at that giant tornado that's k8s.) Who knows, may be they'll conclude job-type is too much of a simplification...

[keith6014]

@kcwong-verseon no one said it was going to be easy. But people said for years the feature will exist. I rather them say "no" so we didn't have to hold our breath. We moved on to Airflow and other custom solutions.

[keith6014]

Are we ever going to implement this feature - 4 years is long time. I think this is an important feature. We are evaluating Nomad as our enterprise orchestrator - but now I am thinking if we are making the right decision to use Nomad.

We stopped offering it internally for this precise reason. Almost 4 years since the issue was raised.

[nugend]

@keith6014 Is Airflow now suitable for service graphs as well as task pipelines? Or were you referring to using Nomad for building task pipelines?

[kcwong-verseon]

@keith6014 I'm sure you know Airflow is a very different beast than Nomad. They have completely different primary objective. They may be able to work together, however.

[keith6014]

@keith6014 Is Airflow now suitable for service graphs as well as task pipelines? Or were you referring to using Nomad for building task pipelines?

for building task pipelines. Data flow, ETL.

[ValFadeev]

Something that might cater for the use case: https://github.com/ValFadeev/rundeck-nomad-plugin Basically, using Rundeck as an enhanced UI for Nomad and taking advantage of its workflow-building and time scheduling functionality. Cons:

• adding Rundeck as another moving piece;
• project needs updating for more recent versions; Pros:
• still running jobs on Nomad;