Open hashibot opened 7 years ago
@paddycarver / @grubernaut / @radeksimko, is this a verified issue/bug? I'm seeing a slightly different issue, but not sure if it's the same as this. If a job has been stopped, or is not running (but the spec has not changed), then the TF nomad provider should update nomad. It might be best to always have the job sent to nomad, and let nomad work out issues with differences in the spec? This issue makes the provider unusable for the most basic use cases.
The underlying issue here also produces the following behavior:
nomad stop
that job manuallyHi @ketzacoatl! I can't say for sure whether this is a verified bug, nor can I explain the behaviour. I'll try to look into this soon and come back with a bit more information, and a fix if necessary. Apologies for the delays on this.
hi @paddycarver, thanks for taking a look! were you able to confirm the behavior we see in practice?
My observation is the same as @ketzacoatl.
It would be just more operator friendly if users could interact via Terraform, rather than nomad status
My versions are Terraform 0.11.5 Nomad (built from master) (0.8.0-dev)
Any updates on this @paddycarver?
Also, a question: Does the Terraform provider always submit the job to nomad, or does it decide whether or not it should?
@paddycarver I took a quick look inside the codebase, and it seems the culprit is:
resourceJobRead
is not implementedmodify_index
wouldn’t pick up changes from the provider. I think “Solution 1” below is a nice way of dealing with it.status
computed metadata to the schema, plus #15 other itemsjob.Stopped()
whether it should be re-registeredplan
plan
now has to read more information before, possibly slowing it down resourceJobRegister
so that it always de-then-re-register, regardless of configuration change or not.plan
with Nomad jobs@paddycarver, I'd love to help resolve this problem, as IMO it prevents serious use of Terraform to manage nomad, do you have any guidance or recommendations on how you would like to address the problem?
cc @katbyte / @radeksimko
@mitchellh I'd be very grateful for your feedback/guidance here.
@cgbaker, With renewed development efforts here, what are your thoughts on this issue?
Hi @ketzacoatl, I wholeheartedly agree that this is something that must be addressed. This shortcoming makes the Nomad provider just about useless for Day 2 operations.
As noted in #15 , there are a few different options for addressing this. The solution partially hinges on whether we want to try to find a general solution that doesn't require modifying and re-releasing the provider every time Nomad releases a new version. On the other hand, with the Nomad product team taking ownership of this TF provider, we can potentially address such a workflow a little better than before. And it may be prudent to find a temporary solution to this issue while we work on a generic/unversioned provider.
The Nomad team is committed to addressing this; I will post an update here soon giving an idea as to the timetable, but my intention is to either resolve this issue as part of the upcoming 1.3.0 version of the provider (targeting Nomad 0.8.x) or the 1.4.0 version (targeting Nomad 0.9.x).
Thank you for your patience and persistence on this issue.
And it may be prudent to find a temporary solution to this issue while we work on a generic/unversioned provider.
That would be great, WRT being able to continue using the provider while future improvements get worked out.
Nomad 0.8.x API support is available in the Nomad v1.3.0 that released today. I will look at finding a longer-term solution for this in upcoming versions; having said that, even if we continue version the Nomad providers, we pledge to be much more responsive in updating the Nomad provider going forward.
The Nomad team is committed to addressing this; I will post an update here soon giving an idea as to the timetable, but my intention is to either resolve this issue as part of the upcoming 1.3.0 version of the provider (targeting Nomad 0.8.x) or the 1.4.0 version (targeting Nomad 0.9.x).
Hello, just checking back up on this - have there been any improvements related to this?
No update as of now, we've been focusing on core Nomad work, finishing the 0.9.0 release.
On Thu, Apr 4, 2019 at 1:02 AM ketzacoatl notifications@github.com wrote:
The Nomad team is committed to addressing this; I will post an update here soon giving an idea as to the timetable, but my intention is to either resolve this issue as part of the upcoming 1.3.0 version of the provider (targeting Nomad 0.8.x) or the 1.4.0 version (targeting Nomad 0.9.x).
Hello, just checking back up on this - have there been any improvements related to this?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/terraform-providers/terraform-provider-nomad/issues/1#issuecomment-479750800, or mute the thread https://github.com/notifications/unsubscribe-auth/ABmPT9wkf1S_J33gI6wrFz0StYLsaQTuks5vdYdQgaJpZM4N5Ctd .
Fair enough, thanks for the update!
Update: this was not resolved in the 1.4.0 release of the provider, but we're still tracking this.
I am stoked to see this on the 1.5 milestone, rock on!
It have been a year since the last comment. Any update on this?
@cgbaker WRT the roadmap, are there refactors or internal changes that block fixing this properly?
we were waiting for the update 2.0 plugin SDK to see if it helped deal with this. we're actively looking at it now; we want this issue dealt with in the next few months as part of the Nomad 1.0 milestone.
Hi everyone,
This issue has been going for a while, but we are still actively tracking it. I also think we are overdue for a more thorough update, so here is a description of the problem and what we've tried so far (and why it didn't work).
This issue happens because the job resource doesn’t have separate attributes for each possible field in the jobspec. Instead, it receives a jobspec as a string (or a path to file that is read into a string), and parses some of the fields into computed attributes.
Since all attributes are computed, Terraform is not able to automatically detect changes, so the provider needs a customized diff function to properly communicate to Terraform if anything has changed. Unfortunately Terraform doesn’t provide the current value stored in the state for computed fields to the diff function, so it can't actually detect changes.
Here are some of approaches we've tried and why they didn't work.
Having the jobspec as the single attribute forces us to do Terraform’s job of computing the diff, which, as described above, we can’t do properly. Having the full jobspec described as the resource schema would be more in-line with the way Terraform works.
But even though both Nomad and Terraform use HCL, the structures supported by each of them are different, thus creating a new syntax for defining jobs that is not compatible with the nomad job run
command nor the Nomad API.
For example, in Nomad a group is defined as:
job "example" {
group "cache" {
...
}
}
In Terraform this job would be declared as:
resource "nomad_job_new" "example" {
name = "example"
group {
name = "cache"
}
}
While having the entire jobspec as the single input for the resource is not ideal, it does provide a nice user experience, where the job is defined only once and can be reused in many different places.
You can check the code for this attempt here.
If we keep the jobspec
attribute but also have the other fields allowed in the job as optional attributes, we could keep the nice user experience of being able to reuse job files but also define some of the values as proper Terraform attributes.
Unfortunately this doesn’t work well because if an attribute is not marked as computed
the provider is not able to set a value for it, so anything that is defined in the jobspec but not in the Terraform resource would not be tracked.
Terraform does allow attributes to be optional
and computed
at the same time. But this falls back to the original issue where the current state value for computed attributes are not available to customized diff functions.
I hope this explanation was helpful, and we will provide more updates as we go. In the meantime feel free to sends us any questions, comments or ideas.
Thanks for the update @lgfa29, it certainly gives clarity on why this is not trivial.
Just curious if you have a current working idea of how this should be addressed or if it's still being explored?
Yes, we are still actively looking for a solution. I will make sure to post new updates, even for failed attempts.
Hi, I've not yet tried it but why not leverage the diff abilities from Nomad to make the diff? Either by exposing it in the Nomad client or adding an endpoint that would return the diff between the new job and the current one (but not actually submitting it).
hi @remilapeyre , we already do that for CustomizeDiff
:
https://github.com/terraform-providers/terraform-provider-nomad/blob/master/nomad/resource_job.go#L518
the issue is that the current Terraform plugin SDK doesn't care about all those computed
fields that we get from the Nomad API... we would need to update the jobspec
. there are some options there, where we use it as a sentinel, which we're looking into. and we've had some discussion with the Terraform plugin SDK team. We are pursuing a few different options, one of which may just be a new, structured nomad_job
resource which doesn't have these problems.
@lgfa29 I like the progress in that PR, and I would love to see that. I have zero qualms as a user with cutting off a major version and breaking a lot of compatibility. We already do this today with the Vault provider where some of our modules depend on older versions and others on newer versions.
I am a BIG! +1 on
resource "nomad_job_new" "example" {
name = "example"
group {
name = "cache"
}
}
I can see how I could modularize this already and produce a terraform module to be consumed upstream from my team with a ton of defaults (e.g.: add sidecars automatically into folks tasks)
Thank you @cgbaker and @lgfa29 for the update. This explains a lot, I had wondered why Nomad's diff features weren't being used here and I did not understand the code itself super well.
Thanks for the feedback @chuckyz. Creating a new resource seems to be the best approach we have right now, so we will move forward with that (but with a better name than nomad_job_new
😄).
Thanks for the feedback @chuckyz. Creating a new resource seems to be the best approach we have right now, so we will move forward with that (but with a better name than
nomad_job_new
smile).
Would it make sense to incorporate it in #149?
Thanks for the feedback @chuckyz. Creating a new resource seems to be the best approach we have right now, so we will move forward with that (but with a better name than
nomad_job_new
smile).Would it make sense to incorporate it in #149?
Yup. #149 is being developed exactly to solve this issue 👍
Is there any plan to release v1.5 soon? Is there more work to finish out this feature? #149 hasn't been touched in almost 2 years.
Hi @eliburke 👋
No plans to get this fixed unfortunately. The work on #149 was a brave attempt to try and map all possible jobspec fields into a Terraform resource schema, but that was not a sustainable approach. As Nomad evolves it becomes almost impossible to manually keep up.
We think that a better approach would be to leverage the Nomad OpenAPI project that is able to auto-generate a Nomad job spec, and the Terraform Provider Framework which is a new and more flexible way to create providers.
But this will take a significant effort that is not in our roadmap yet.
suggestion: to have a quick workaround for this problem, what we have been doing for quite some time is the we render out the entire Nomad job file and use it via the terrafrm-nomad provider (in comple rendered form)
What this does is, when you suspect things, and want to check what is up with the job, you can try out nomad job plan myjob.nomad
instead of the usual terraform plan
.
In my opinion this provides an easy escape hatch mechanism.
Can the Submission field in the API now be used? enable support for setting original job source
Hi @tristanmorgan,
Unfortunately that's not enough. The key issue here is describing the Nomad job specification as a Terraform resource schema and detecting changes on individual fields. The resource already has the raw jobspec stored, which would be the same as the value returned by the new job submission endpoint.
Just made a comment on https://github.com/hashicorp/terraform-provider-nomad/pull/238#issuecomment-1833877526 about this.
Reading @jorgemarey comment again, I noticed this part:
We detect changes by having a shasum of the rendered job and seeing if it changes with the value stored for the previous resource version.
While not quite related to this issue, it made me realize that we can use the job submission data (if available) to detect changes to jobspec
, which should help mitigate drift problems like this when the submission data is available.
Is this improved at all with v2 of the provider? I need to take over control of previously hand-generated Nomad job templates with Terraform. Usually when I do this workflow I create a resource in the Terraform code, import the existing resource, and reconcile the differences. However, the fact that this provider doesn't detect differences completely breaks that workflow.
I briefly tried upgrading to v2 provider but still got a unexpectedly large diff, how much of that was this bug and how much was v1 to v2 incompatibilities I didn't determine. I dropped back to v1 to not burn time upgrading to a new version that might have the same major bug.
Hey folks! @gulducat and I did a quick re-assessment of this bug and here's what the current situation is:
terraform plan
will successfully detect changes that are made via nomad job run
and the Nomad UI.nomad job scale
or via the Nomad API directly.nomad job stop
.After some internal discussion I'm marking this for further roadmapping. We'll update again once we know more.
@tgross awesome update, thank you for all that info. Even with a big lift in the future, smaller/incremental improvements are very welcome!
This issue was originally opened by @blalor as hashicorp/terraform#14038. It was migrated here as part of the provider split. The original body of the issue is below.
Terraform Version
Terraform v0.9.3
Affected Resource(s)
Terraform Configuration Files
Debug Output
Log for 2nd
terraform apply
: apply.log.txtConsole output:
Expected Behavior
Terraform should have noticed that the job had changed in the cluster and updated it.
Actual Behavior
Nuttin', honey.
Steps to Reproduce
nomad agent -dev
terraform apply
nomad status example
shows one allocation for task groupgrp
change task group count:
nomad status example
shows two allocations forexample.grp
terraform plan
shows no changeterraform apply
makes no changenomad status example
still shows two allocations