Allow output helper to read values from a different environment

[leogargu]

Summary

Allow configuring the output helper to read the outputs of a stack in an arbitrary environment, not necessarily the current one.

In other words, provide a similar functionality as mock values for the output helper, but dynamic.

Motivation

The output helper can be used in a stack to read the outputs from another stack that has been deployed in the same environment (same TS_ENV).

If the second stack has not been applied, the only option currently is to supply mock values. In my experience, these mock values are only useful if they are actually taken from a working, applied environment (e.g. "DEV"), that can be used as reference/fall back. After applying the reference stack I can copy and paste the output values as mocks into the output helper.

The downside is that the values defined as mocks are static: if the reference environment changes, they don't get updated, which causes misleading errors while developers are working on their own environments, resulting in time wasted.

So it would be very useful to be able to configure the output helper to read the outputs from a different environment, for example:

vpc_id = <%= output('vpc.vpc_id', default-env: 'DEV') %>

Drawbacks

None I can think of! It's additional functionality to better support what to me is a common workflow

Unresolved Questions

I think there are two options to do this, one may be simpler than the other to implement but both would solve the current problem:

• Define a default/fall-back environment that the output helper will read the values from if the dependent stack has not been applied to the current environment
• Allow the output helper to define the environment to read values from, regardless of whether it's been applied or not

[ricochet1k]

More generally, it would be really helpful if output could be targeted at anything in the layer stack. Like for example, if one stack wants to use TS_EXTRA, but wants to refer to output from another stack that doesn't use TS_EXTRA.

[alexjfisher]

More generally, it would be really helpful if output could be targeted at anything in the layer stack. Like for example, if one stack wants to use TS_EXTRA, but wants to refer to output from another stack that doesn't use TS_EXTRA.

@tongueroo Is there a solution to this? I think I'm trying to achieve exactly the same thing. eg. Have a VPC stack that doesn't use TS_EXTRA, but an application stack that does.

[tongueroo]

There's currently no way to do this.

Last time, I took a look at this, it was pretty complex and shelved it for the sake of time. I would really like terraspace to be able to do this, though.

Some of the reasons for the complexity. Terraspace boots and memoizes info like Terraspace.env and Terraspace.extra. This is because these values are globally used as part of the "terraspace run". Terraspace use threads to do some parallelization and changing the values screws up the threading. There's already some thread-safe issues that will probably have also be handled with a probably mutex lock.

Got some ideas. Will eventually make it happen. Again, really want this feature.

[leogargu]

Hi @tongueroo thanks for the context! I've been wondering whether it'd be possible to write a custom helper that would get an output from the dev environment. Would something like this be feasible at all, or does the env variable+multithreading situation get in the way regardless? (I've been trying to get this to work but tbh my ruby isn't up to scratch 🙈 )

In config/helpers/custom_helper.rb:

module Terraspace::Project::CustomHelper
    def get_from_dev(stack_name, output_identifier)
        # magic here to get the stack_name.output_identifier from dev
    end
end

And then: vpc_id = <%= output('vpc.vpc_id', mock: get_from_dev('vpc', 'vpc_id')) %>

[leogargu]

Following up on my previous comment, I've been having a play and the helper below seems to work fine so far, assuming that:

1. The dev environment has been deployed previously
2. Terraspace has previously built the files in the cache

Which are reasonable assumptions for me. I don't know how well it will stand the test of time though 🤔- any thoughts or words of caution would be most appreciated!

def is_ephemeral(env)
  persistent_envs = ["dev", "stag", "prod"]
  return !persistent_envs.include?(env)
end

module Terraspace::Project::CustomHelper
  def get_from_dev(stack_name, output)
    Terraspace.logger.debug "Helper get_from_dev(#{stack_name},#{output}) called"
    aws = AwsData.new
    stack_dir = ".terraspace-cache/#{aws.region}/dev/stacks/#{stack_name}"

    # Exit early if we're not dealing with an ephemeral stack - we don't need
    # to provide dev values in this case
    if !is_ephemeral(Terraspace.env)
      Terraspace.logger.debug "Skipping retrieval of mock values from dev because we are targetting a persistent environment: #{Terraspace.env}"
      return nil
    end
    if !Dir.exist?(stack_dir)
      Terraspace.logger.debug "Stack #{stack_name} has not been built and/or deployed in dev"
      return nil
    end
    # Attempt to retrieve the output value from the ts_dev environment
    output_value = `terraform -chdir=#{stack_dir} output -json | jq -r .#{output}.value`
    Terraspace.logger.debug "Output value retrieved: #{output_value}"
    return output_value.strip
  end
end