Library Variable Templates VaryByEnvironment

[michaelnoonan]

It is common to define environment-scoped variables that are shared by multiple projects in Library Variable Sets, but we don't provide an equivalent for Variable Templates. I can easily see circumstances where you'd want a shared tenant-environment scoped variable - think shared Azure Storage Accounts, Amazon S3 buckets, a common environment-specific database.

We should allow users to configure Tenant Variable Templates defined in Library Variable Sets to choose whether they are Constant or if a different value should be provided Per-Environment. The tenant variable screen could show an additional tab for Environment Variables in addition the existing Common Variables and Project Variables for editing these variable values.

Unfortunately we would have to consider how to handle someone changing the configuration of a Variable Template, should we veto the change if there are any defined values?

Source: http://help.octopusdeploy.com/discussions/questions/9008-variable-templates-no-option-for-environmentscope-selection

[vitaliitylyk]

Hello, do you have any ETA when this will be implemented?

[Slowacki]

Wow, this feature would be plenty useful in my current project. Are there any plans to implement this in the nearest future?

[octaviobffernandes]

In my project as well.. would love to see such a feature like this

[olhaburlai]

That's really what need Hope it will be implemented soon

[remmedios]

It sounds great, would be very useful for me too. Do you plan to put it through?

[christopherbarr]

I'm the original author of the referenced support forum post... really wishing this feature would be included in an upcoming release - I still can't use multi-tenant option of Octopus due to this.

[michaelnoonan]

Hi everyone, thanks for commenting on this issue! We are about to undertake some work where we think some extra options for variable templates will help. Keep you eyes on our blog for an RFC which will go into more detail.

Some early thinking is that we could make variable templates more "self-contained" and allow each variable template to express:

• which Octopus entity should define the value(s). eg; Tenant, Environment, Deployment Target
• how many value permutations should be required. eg; Should each Tenant provide a single value (like "common variable templates" today) or a value per-Project+Environment (like "project variable templates today") or a value per-Environment (as we suggest in this issue).

This would enable you to do things like:

1. Define a Variable Template where each Environment targeted by a Project's Lifecycle(s) should provide a value (essentially becoming a value per-Environment)
2. Define a Variable Template where each Deployment Target targeted by a Project should provide a value (something like "The temporary drive should be D:\ on this machine")
3. Define a Variable Template where each Tenant connected to a Project should provide a value once-per-Environment regardless of which projects it is connected to (the core reason for this issue)

At this point the main driver is to target the variety of template configurations you could define for Tenants and Environments. Deployment Targets can get complicated because they are calculated at Deployment time, and it is difficult to calculate them beforehand.

What are your thoughts?

[chrisjbarr]

Can't wait... really need option 3. =] So excited for you to guys to finally start exploring this option, this will make deployments for multi-environment tenants so much easier.

[fatmanmclone90]

+1 for number 3

[kneumei]

How would I handle the following situation.

Two Environments defined:

1. "Simple Environment" with one machine (web and service on same machine)
2. "Complex Environment" with two machines (web machine, service machine)

I have two tenants:

1. Tenant A => "Simple Environment"
2. Tenant B => "Complex Environment"

I have a variable called "Machine Name"

If I create a template per environment, I get to supply one value. This works for Simple , but not for complex.

Currently with "normal" variables I can scope by environment + machine name to address this issue.

It would be nice if for a single variable template, I could provide different values for different scopes.

[michaelnoonan]

I would like to see this kind of thing happen too, but it hasn't come to the top of the "priority-vs-complexity" list. At this point it we think it will be implemented as a first-class citizen in remote release promotions: https://octopus.com/blog/remote-release-promotions-rfc

[michaelnoonan]

Whoops, closed by accident. Wrong keyboard shortcut!

[Vacant0mens]

Another +1 for number 3 :)

[Vacant0mens]

@michaelnoonan, is there a projected version release for the scoped-variable-set-templates yet? The last bit of the release promotions blog post says it'll be released with 4.0, but that's already released, and I don't see the option to scope variable set templates.

[michaelnoonan]

Hi @Vacant0mens. We are determined to do this as a larger piece of work unifying our variable system across the board. This issue is the placeholder I could find for it: https://github.com/OctopusDeploy/Issues/issues/4160

You'll see a link to a google doc which will become a spec in our https://github.com/OctopusDeploy/specs repository soon.

I'll post back in this thread when I have more details.

[markgould]

Hi @michaelnoonan any updates on this yet? I think a lot of people would find this useful.

There is also a uservoice request tracking it https://octopusdeploy.uservoice.com/forums/170787-general/suggestions/33475216-set-tenant-common-variables-per-environment

[michaelnoonan]

Hi @markgould unfortunatly no update yet. We agree on principle - it's just a matter of timing.

This issue is a central point the team are using for this larger exercise: https://github.com/OctopusDeploy/Issues/issues/4390 Take a look at the Google Document which is a starting place for the whole shooting match.

[markgould]

Thanks, I'll read through it all!

[dave-sampson13]

Hi, just checking in on whether you have an expected time frame for this feature? OD has some of the best config data management available & this would top it off! Thanks!

[dave-sampson13]

Any update on how this is progressing / ETA?

[michaelnoonan]

Hi @dave-sampson13 👋 I agree. Configuration management in Octopus is a critical and powerful feature. It's also used by EVERYONE.

We have a good appetite to invest in the next steps here, but it hasn't reached the top of our priorities this year. I am confident we will address this gap, but I cannot offer insight into timing. Sorry.

[chrisjbarr]

So sad that after all these years my original request still hasn't gotten shipped..... hopefully 2020 is the year! :)

[nathangiuliani]

This is a pretty big deal for us, leading to values needing to be entered per-package (scoped as appropriate) when they should simply be entered once per-tenant, per-environment.

How are others working around this?

[boudomi1]

This would be a game changer for our teams as well as we need to define variables that are used by tenant/by environment but are shared with multiple projects. I really hope we will be able to see this come to reality somewhere in 2020!

[DannyArr]

We're currently in the position of having to duplicate the environments. A new set of environments for each tenant :( Very sad.

[dgard1981]

I've recently started using tenants, and unless I'm missing something I still only have two options for variables -

• Project Variables: per project, per environment.
• Common Variables: every project, every environment.

What is sorely lacking (IMHO) is -

• Environment Variables: every project, per environment.

For example, say you have the following 2x environments, 1x variable and 10x projects. If you want to set a value for that variable that is the same across all projects but different in the two environments, you'd have to set 20 variables in the tenant settings. What if it's 25 projects, or 50, or 250? And what if you have 30 variables, or 50, or 250? And what about if you had 5 environments, or 10, or 15?

Example - 25 projects, 4 variables, 5 environments = 500 places (Project Variables) to set the 20 different values. If an option like Environment Variables were available for tenants, that could be cut to 20 places to set the 20 different values.

My point is that it's very easy to make mistakes when you have to manage the same value across multiple projects. Not to mention the overhead of having to add/maintain variables to every project template in the first place.

If we had something like the Environment Variables idea discussed here my life (and I suspect many other peoples lives) would be infinitely easier. I'd go as far as to suggest that uptake of Tenants would increase with better variable management.

This ticket has been open for 4.5 years now, so it would be good to have a resolution either one way or another.

Thanks.

[Vacant0mens]

@dgard1981 and @christopherbarr one way I've gotten around this (until the development for this issue is resolved) is creating a Common Template for each environment, and then a Library Set Variable with values referencing each Common Template scoped for the applicable environment. It's a little bit hacky/janky, but it works for now.

Until the Variable Unification project changes things, the main complication that I can see with this particular issue (@michaelnoonan, or anyone else, correct me if I'm wrong) is that there is a high chance that not all projects follow the same lifecycle, which would make them have different environments that they deploy to. For example,

Project-1 -> Lifecycle-A -> Environments R, S, T, and Q
Project-2 -> Lifecycle-B -> Environments X, Y, Q, and Z
Tenant-1 -> Project-1, Project-2

So here you potentially have up to 7 values (or 8 depending on how it's programmed) for your one varied-by-environment Common Template, and that's not even accounting for what environments the tenant is connected to those projects with (it may not be all of them). And then adding yet another project for that tenant that has a different lifecycle, what do you update (variable attributes? tenant attributes? or...?) to know that it needs more values? And do you have an option to group environments to avoid duplicate values? (@michaelnoonan if your teams have these questions answered/designed-out, I'm curious how you solved it.)

So the more lifecycles you have that are different, the more potential values you'd have based on the projects and environments each tenant is attached to, and the more processing (and more calls back to the Api) it'd take to figure out if you have all the values covered or not. Not only does this seem to be fairly difficult to program, but it also potentially complicates the database a bit as well. The relationship between Projects, Variable Sets/Templates, and Tenants is already fairly complicated (programmatically), and this is a whole other ball of wax, so to speak.

I'm 100% in favor of this feature and excited to see it be released, but I can definitely understand why it's taking so long, especially with the Variable Unification project.

[dgard1981]

@Vacant0mens, thanks for the tip.

One thing I am considering for now is using conditional variable substitution. However, this is not easy on the eyes or brain, and again is likely to be a breeding ground for messing things up and ruining peoples weekends when stuff blows up due to mistakes.

For example, with just three environments it already looks horrible.

#{if Octopus.Environment.Name == "BVT"}some value#{/if}#{if Octopus.Environment.Name == "UAT"}some other value#{/if}#{if Octopus.Environment.Name == "PRD"}real value for real people#{/if}

[markgould]

@dgard1981 and @christopherbarr one way I've gotten around this (until the development for this issue is resolved) is creating a Common Template for each environment, and then a Library Set Variable with values referencing each Common Template scoped for the applicable environment. It's a little bit hacky/janky, but it works for now.

Until the Variable Unification project changes things, the main complication that I can see with this particular issue (@michaelnoonan, or anyone else, correct me if I'm wrong) is that there is a high chance that not all projects follow the same lifecycle, which would make them have different environments that they deploy to. For example,

Project-1 -> Lifecycle-A -> Environments R, S, T, and Q
Project-2 -> Lifecycle-B -> Environments X, Y, Q, and Z
Tenant-1 -> Project-1, Project-2

So here you potentially have up to 7 values (or 8 depending on how it's programmed) for your one varied-by-environment Common Template, and that's not even accounting for what environments the tenant is connected to those projects with (it may not be all of them). And then adding yet another project for that tenant that has a different lifecycle, what do you update (variable attributes? tenant attributes? or...?) to know that it needs more values? And do you have an option to group environments to avoid duplicate values? (@michaelnoonan if your teams have these questions answered/designed-out, I'm curious how you solved it.)

So the more lifecycles you have that are different, the more potential values you'd have based on the projects and environments each tenant is attached to, and the more processing (and more calls back to the Api) it'd take to figure out if you have all the values covered or not. Not only does this seem to be fairly difficult to program, but it also potentially complicates the database a bit as well. The relationship between Projects, Variable Sets/Templates, and Tenants is already fairly complicated (programmatically), and this is a whole other ball of wax, so to speak.

I'm 100% in favor of this feature and excited to see it be released, but I can definitely understand why it's taking so long, especially with the Variable Unification project.

I do the same as you, but we currently only have 3 environments, so it's not so bad. It is possible to leverage the Octopus API (I did a POC using the C# library) to automate this somewhat, but it's not ideal.

[Vacant0mens]

@Vacant0mens, thanks for the tip.

One thing I am considering for now is using conditional variable substitution. However, this is not easy on the eyes or brain, and again is likely to be a breeding ground for messing things up and ruining peoples weekends when stuff blows up due to mistakes.

For example, with just three environments it already looks horrible.

#{if Octopus.Environment.Name == "BVT"}some value#{/if}#{if Octopus.Environment.Name == "UAT"}some other value#{/if}#{if Octopus.Environment.Name == "PRD"}real value for real people#{/if}

Yeah, you'd be better off with my suggestion. Having to update that formula for each tenant wouldn't be great either.

[fbkevy]

Hi, any ETA on this feature? I could really do with it :-)

[markgould]

Just bumping this back up to see when we can expect to see some movement on this? Having to do the above steps and go thru clunky workarounds is frustrating.

[bdway]

Same comments as those above, this is highly valuable functionality. Without it we are forced to into hacky work arounds. For us it doesn't even make sense to utilize Tenants and have to manage the same variable value in multiple places. We have opted to create a large number of environments which is messy, but allows for managing a single value in a single variable set.