search

hyperbadger / nomad-pipeline

Run pipeline-style workloads in Nomad
MIT License
41 stars 2 forks source link

Doesn't work with parameterized jobs #9

Open waquidvp opened 2 years ago

waquidvp commented 2 years ago

Nomad pipeline fails at the "init" stage when used in a parameterized job. The following error is returned by Nomad: cannot update non-parameterized job to being parameterized.

This is an issue with Nomad in which it currently doesn't support updating dispatched jobs. Nomad pipeline relies on updating the job in order to control the count parameter of each task group.

See the following issue already raised in Nomad: https://github.com/hashicorp/nomad/issues/10941.

muuki88 commented 1 year ago

Hi @waquidvp

We are evaluating nomad-pipeline and it seems the issue you opened has already been closed. Does this mean that updating parameterized jobs works? Or is this something that needs to be fixed in this very plugin?

Thanks for you work on this :heart:

waquidvp commented 1 year ago

Hey @muuki88,

Thanks for taking at look the project. Like you said, the upstream Nomad ticket was closed and unfortunately the conclusion was that the fix wouldn't go in. However, there are two possible workarounds:

1. Use a patched version of Nomad

This is my current method, I have been maintaining a patch file which I have been applying when new versions are released (currently 1.4.3). The patch is derived from the pull request that was closed. This option isn't sustainable but it's what I have been using in our org. It adds some overhead when new versions of Nomad comes out as it requires you to maintain your own Nomad binaries. If you are interested in this option, I have attached the patch I use and I can provide more complete instructions in this thread - if needed.

update-payload-on-dispatched-job.patch

2. Using the nomad-pipeline API to dispatch jobs

This functionality is still work in progress but will be ready to use soon. As part of this feature, there will be a server component to nomad-pipeline which exposes an API. The way it works is by copying the job definition from a parameterised job and creating a new standard job using the same definition. This does however have some limitations which I am still working on fixing, the main one being the lack of support for the dispatch payload. Here is the WIP branch: https://github.com/hyperbadger/nomad-pipeline/tree/f-dispatch-pipelines.