SSM Dynamic Reference latest version

Ricapar commented 5 years ago

1. AWS::CloudFormation::Stack-SSM Dynamic Referecnes

2. Scope of request

As documented here, CloudFormation supports resolving values out of SSM using the {{resolve:ssm:MyParamNameHere:1}} notation.

It is also documented that "You cannot currently specify that AWS CloudFormation use the latest version of a parameter."

I am requesting that we be able to support using the latest version of a parameter.

3. Expected behavior

This currently does not work:

{{resolve:ssm:MyParamNameHere:latest}}

Expected behavior: When something similar to the sample above is used within CloudFormation, it is able to resolve and use the latest value.

4. Suggest specific test cases

When :latest is used as the version, it retrieves and uses the latest value from the SSM Parameter.
If a Stack Update is done, CloudFormation re-checks the value and performs stack updates as necessary.

5. Helpful Links to speed up research and evaluation

AWS Documentation on Dynamic References: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html

6. Category

Management (CloudTrail, Config...) (CloudFormation)

kastork commented 5 years ago

It seems also to be the case that the length validation of MasterUsername and MasterUserPassword is applied before the '{{resolve...}}' is resolved, so if your expression is longer than 41 (or 63) characters the secret is never looked up.

https://forums.aws.amazon.com/post!reply.jspa?threadID=299271

mikerochip commented 5 years ago

I feel like leaving off the version entirely should be an option. That would also make it consistent with the syntax for dynamic secret resolution.

ngamradt-turner commented 4 years ago

AWS folks, do we know if this is going to get on the roadmap soon?

robot-apocalypse commented 4 years ago

This is a huge pain for my use case, and I think it's a decent one. In order to dynamically reference a parameter such as "\<env-name>/params/timeout", I have to either construct the parameter name in bash or some other language and pass it in as a parameter to the template, or use these "dynamic" references which can be constructed using sub or join. However, by forcing us to hardcode the version number, I have to keep track of and update any references to the parameter manually. I strongly object to calling these "dynamic" references since they are actually much less dynamic since you have to specify the version, at least to the end user.

I'm actually having a hard time understanding the utility of specifying the version number. Wouldn't the above use case be the usual reason for dynamically constructing the parameter name? Why is this dynamic behavior so different than the non-dynamic version, which doesn't force a version number?

parsnips commented 4 years ago

I'm actually having a hard time understanding the utility of specifying the version number. Wouldn't the above use case be the usual reason for dynamically constructing the parameter name? Why is this dynamic behavior so different than the non-dynamic version, which doesn't force a version number?

It's most likely not an end user utility thing, but a cloudformation leaking it's implementation. My guess is that the template needs to resolve deterministically (for eg the changeset functionality), and there's not a straightforward way to resolve the latest parameter from a prior run to an actual version (without building a transform).

mikerochip commented 4 years ago

I'm actually having a hard time understanding the utility of specifying the version number. Wouldn't the above use case be the usual reason for dynamically constructing the parameter name? Why is this dynamic behavior so different than the non-dynamic version, which doesn't force a version number?

It's most likely not an end user utility thing, but a cloudformation leaking it's implementation. My guess is that the template needs to resolve deterministically (for eg the changeset functionality), and there's not a straightforward way to resolve the latest parameter from a prior run to an actual version (without building a transform).

Yeah I'm pretty sure you hit the nail on the head here. Case in point: secrets DO let you leave off the version and use the latest, and they suffer from the exact problem you described.

I put in a feature request to force a resolve via a flag or capability or similar for secrets during change set creation: https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/369

Juberstine commented 4 years ago

I would love it if there was a way to use Latest. Following Issue.

chriscloud27 commented 4 years ago

Great idea! This feature would make life much easier

kirnberger1980 commented 4 years ago

Please implement this feature. We definitely need this for our CF Templates as we use SSM Parameters a lot.

goyertp commented 4 years ago

Use of latest is often necessary. Great Idea!

mykyta-kharchenko commented 4 years ago

Folks, I am moaning already. Pls add the support of :latest ASAP

joaquin386 commented 4 years ago

Best enhancement for our systems.

benkehoe commented 4 years ago

I'm going to go against the grain and say that while I want referencing parameters to be easier, I don't want :latest, at least until there are some larger changes in CloudFormation. As pointed out by @mikerochip, you could get the parameter resolved when you deployed the template, but there's no way to subsequently update it, because your template isn't changing. And of course, your template is not longer deterministic, which is less than ideal. An argument could be made that CloudFormation could watch that parameter for changes and redeploy automatically, but I think that's a recipe for surprises, and I hate surprises in my operations :-)

So what I want is some way of starting with the name of a parameter, and along the way the latest version gets baked into the deployment. So maybe what I'd rather have is some way of having a map of the parameter->version, and be able to re-resolve that mapping to perform an update. So maybe it's like, in my template I reference a parameter by name without a value, and when I'm using CreateChangeset, it snaps the parameter->version mapping using the latest values, or I can override the versions if I want. So now, I'm still fully specifying the versions of the parameters, but because it's outside my template proper, it's less gross in my source control.

clayvan commented 4 years ago

A good example of why this is so important:

AWS says to use nested stacks where possible
Nested stacks don't have visibility into their changes via changesets
Okay so I need to break my nested stacks into individual stacks because I need changeset visibility
So i'll use Exports for outputs to pass values to dependent stacks
But if that Export value changes, dependent stacks must first be deleted.
Okay so AWS recommends storing the values in the SSM parameter store.
But to reference that SSM Parameter in my dependent stack, I need to know the version of the parameter, which I don't know without passing the version into my dependent stack somehow? Which was the whole point of using the SSM Parameter store in the first place?
Okay well maybe I can use some wrapper orchestration to grab the Parameter Version when I create it, oh the AWS::SSM::Parameter resource doesn't even have a GetAtt for version.

This seems like a big gap in the logic here.

However if you take a look at Terraform, yet again we will see that Terraform is doing it the "right" way.

https://www.terraform.io/docs/providers/aws/d/ssm_parameter.html

Terraform imports the data from a SSM parameter without caring about the version.

I keep trying to work around the CloudFormation limitations because I really want it to work for me, but this might be the final nail in the coffin for me to switch to Terraform.

ambsw-technology commented 4 years ago

Without a bulk update option for Parameter Store, a stack literally cannot watch parameters and mutate dynamically. What if I change my Fargate CPU size to something incompatible with my Memory size and it tries to update before I can modify the Memory size as well? That's a pandora's box. So we're stuck with manually-triggered updates and, if so, why not the manual event (i.e. CF change) that we're already used to.

So that brings us back to the original issue of a dynamic reference that requires a version. By definition, for n dynamic references, I either have to hardcode or pass (e.g. as parameters) n version numbers. How is that better than looking up and passing the values?

The only place I can even imagine a net-improvement is for secure parameters. My deployer no longer needs permission to decrypt them (just to see the latest version). On the surface this certainly seems more secure, but here's the catch... my deployer can also roll back secure values to any historical value they like (including known/breached values). That's a huge security hole and, ironically, one best addressed by forcing them to use the latest parameter!

EDIT: I now see that there's also a quirk in the way nested stacks are resolved during updates. If the dynamic reference is not part of the top-level stack, something must be changed in a nested stack to get them to update. A version number would be one way of prompting that change. Since we already have solutions for this problem, there's nothing special about a version number except that it would naturally cascade to the exact right scope.

luiseduardocolon commented 4 years ago

I've been following this thread, and it seems like the discussion has gone back and forth in terms of a) whether to provide {{resolve:ssm:MyParamNameHere:latest}} or not, b) how to provide it, and c) what is the side effect in nested stacks (using dynamic references in root, child, etc.). Would it be ok to ask one of you to refresh the requested requirements to cover all these issues, and then get consensus? Once we collectively do that, I can have someone internally give me (us) a sense of what we can do and when. Sound fair to you?

mfriesen commented 4 years ago

I personally like "{{resolve:ssm:MyParamNameHere:latest}}". The implementation seems straight forward to me, where the "latest" version is calculated for each parameter. Then the template(s) are rewritten where "latest" is changed to the correct version number. Of course, maybe I'm missing some use cases?

ambsw-technology commented 4 years ago

I can't claim any special authority here, but I've thought about it recently so I'll make an effort to outline some items (enumerated for easier reference).

1) Use Cases: Here are some CloudFormation use cases (with and without dynamic references) for consideration:

a) I have a local source-of-truth for parameter values
b) I have a local source-of-truth for version numbers
c) I use the API to load parameter values and pass them to the stack
d) I use the API to load latest version numbers and pass them to the stack

2) Problems: Problems with current implementation

a) If we must specify a version number, dynamic references are just as much work as passing raw value. We must hard code or provide every single version.
b) While the indirection of a version number prevents some data from passing through a local system, it's not clear that it significantly increases security. A deployer can simply roll back to a previous version. Template owners could increment minimum values, but we're almost back to hard coding them.

3) Proposed Solutions: All of these are semantic variants on a latest behavior:

a) Add a latest tag to SSM
b) Support an empty/absent version as latest. This is consistent with dynamic secret resolution.
c) Use a latest tag here and introduce a latest tag to dynamic secrets.
d) Support both latest and empty for both secrets and SSM

4) Additional Considerations:

a) Automatic Updates: CloudFormation template cannot respond in real time to changes in SSM parameters. Since the CLI (and UI) have no way to bulk update, this can result in inconsistent/incompatible parameter values.
b) Update on Stack Change: This is the timing/pattern for the existing parameters. It is an acceptable time for the change to occur. Inconsistent values are still possible e.g. due to a concurrent change to SSM during deploy, but the only way to avoid this is to keep a local source of truth of some kind (parameters or versions) and that introduces versioning issues.
c) Nested Stacks, Current Behavior: In this version, the latest tag is only supported on the top-level stack where it is always resolved. If you want changes to cascade, you pass these top level parameters to nested stacks. They update as they would have if you had changed an actual parameter value. Version-based parameters can be hard-coded into nested stacks since they cannot be changed without triggering a stack change anyway.
d) Nested Stacks, Cascade Update: In this version of the proposal, the entire tree of stacks must be walked every time to resolve dynamic references. This ensures updates are propagated. The parameter space is simplified because the latest tag does not need to be passed from the top-level to deeper stacks.

My Conclusion

Since 4b and 4c most closely mirror the current parameter behavior, they are an acceptable first step (and likely require little effort). I see the merits of 2b but believe that explicit is better so I would pay the cost of "two ways to do a thing" to go with 2d (or 2c). Other changes may be worth considering, but they aren't necessary to introduce latest and can be discussed/proposed as improvements that need to be justified by their own use cases.

Ricapar commented 4 years ago

When I submitted my original post back in 2019, I was already aware of some of the issues that'd result of a concept of a :latest tag resolution, namely around the deterministic nature of how CloudFormation operates.

This is most obvious when you're using custom resources - if you don't change any of the parameters that are going to it, CloudFormation doesn't even look at the resource. And of course - why should it? No parameters have changed, so how could the resource possibly need any updating?

I'm taking a step back and thinking about why I and others are asking for this feature: We need a place to hold common variables across CloudFormation templates. A place to hold a source of truth so we don't have to enter in the same data into parameters every time.

I do know that Parameters are evaluated every time a stack is run, including when the parameters are references to Parameter Store.

For example, we commonly use this to grab the latest Amazon Linux AMI:

  LatestAmiId:
    Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
    Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2'

If I launch a stack with that today, it finds today's current value. If I come back a month or two later, I can expect that parameter may resolve to something different and I'd get back a different AMI ID.

This part works great when there's one parameter you're sourcing - but becomes a lot more tricky when you have many more. You can quickly hit your maximum parameters, and you also end up just cluttering the interface to your template. Which parameters do I actually expect future developers to fill in? What should they leave alone?

Even moreso, if you're exposing CloudFormation through Service Catalog, then you're going to bombard the users with a list of ugly looking parameters that serve no purpose to them at best, and at worst will confuse a not-as-technical user.

So maybe what CloudFormation needs isn't integrated resolution of SSM Parameters - maybe what we need is a way to set variables and constants, whose values can be derived from SSM Parameters or just hard-coded into the template. There are a lot of cases where I don't want to repeat myself 100x in a template, but I also don't necessarily want that piece of information in a parameter.

I feel like something like the mock-up example below would solve the SSM resolution issue, as well as make life a lot easier for a lot of people who have tons of parameters on their stacks because that's the only way to not repeat a value everywhere in your code.

If the Variables section behaved the same way that the AWS::SSM::Parameter::Value<> CFN Parameter type worked then I would think it wouldn't introduce any new issues with deterministic behavior during template execution (at least, not any new ones!).

---
Parameters:

  ResourceDescription:
    Type: String

Variables:

  OwnerTag:
    Type: String
    Value: "John Smith"

  PrimaryVPC:
    Type: String
    ValueFrom: !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/mycorp/vpc"

Resources:

  MyResource:
    Type: AWS::SomeService::SomeResource
    Properties:
      VpcId: !Ref PrimaryVPC
      Description: !Ref ResourceDescription
      Tags:
        - Key: Owner
          Value: !Ref OwnerTag

mikerochip commented 4 years ago

I think this solution is great @Ricapar.

I've used maps as a ghetto workaround for the lack of ergonomic constants.

The only workaround I can think of right now for lack of variables is to have a stack purely composed of parameters (with SSM::Parameter types) and maps (for constants) which then exports all of those parameters and map values, then all the other stacks could import those values, which feels super gross. This variable design feels much better.

I think the expectation for the user is that (non-const) variables need to be resolved every time a change set is created even if nothing else about the stack changes. This way we can avoid a mass number of export/import - we could have variables be populated via dynamic ssm parameters or imports.

ambsw-technology commented 4 years ago

I think @Ricapar's concern deserves discussion, but it's now completely unrelated to :latest and really should be a separate thread.

If the problem is just cluttering a parameter space, it seems like a NoEcho-style flag (that doesn't display the parameter at all) would be adequate.
If there's also a problem with the number of parameters, I'd want to understand the use case. Is a template with that many parameters the right solution or is it the byproduct of a different architectural root cause that really could/should be fixed?

For example @mikerochip mentions exports as a possible (if undesirable) solution. @clayvan also mentions that the use of exports is blocked (or at least discouraged) by the fact that dependent stacks need deleted to make changes. We're also limited by this behavior.

We use two related libraries of CF template (aws-cf-templates and cfn-modules) that make heavy use of exports to support dependency injection. For example, there's a CF template to create an ECS Cluster stack and a CF template for an ECS Service stack that accepts the stack name of the Cluster (getting cluster information from exports like {$ClusterStack}-<Variable>). If we could cascade export changes to dependent stacks (you already resolve that list to block the update), it would make the pattern viable for more complex cases.

PenelopeFudd commented 4 years ago

I'm in favor of :latest or just leaving off the version.

If the parameter is a ssm-secure parameter '{{resolve:ssm-secure:parameter-name:version}}', it's probably credentials, and requiring a fixed version kills credential rotation
If these dynamic references are equivalent to variables, it's rather odd to have constant variables. :-)

Juberstine commented 4 years ago

Any movement on this issue?

mikerochip commented 4 years ago

It's going to be pretty difficult, maybe impossible, to get a consensus here.

I think the best path forward is to rethink the problem and refactor the solution, which was basically what @Ricapar suggested with his variables proposal, but there are many ways to solve the problem.

The root problem here is that CloudFormation needs a better DX for injecting external information into a template from a data source when the data sources don't change but the values from the data sources do. Cfn's built-in parameter system doesn't solve this problem well. I like that feature, I think it's great because it has no dependencies on external systems and is easy to use. It's just not a sufficient solution in the aforementioned use case.

I hesitate to suggest a solution because I fear it will become a distraction and derail this thread again. Instead, I'll put focus back on the problem and where we're at currently.

In my mind, dynamic resolution of secrets and ssm parameters is a problematic solution for these reasons:

It's dependent on secrets/ssm, which is weird because secrets and ssm params are infra, and you're suppose to use cfn to provision infra. It's sort of like a cyclic dependency, but not really. That makes it really awkward. Your best option is to manually provision secrets and ssm params so you can use them as dynamic references. That's weird! Kind of defeats the purpose of cfn.
As this entire thread highlights, the feature's pretty difficult to use as is, without a "latest"
Customers like us are experiencing some real pain and because of the dependency on ssm and secrets, we are giving confusing feedback. We keep asking for a "latest" when dynamic references may not be a good solve in the first place.
The CloudFormation team isn't getting useful, actionable feedback. They keep hearing complaints about the lack of "latest." They keep hearing conflicting feedback. The current solution is creating this stalemate situation. It looks like customers don't know what they want and that leads to inertia and a lack of clarity on the dev team, so they'd rather focus on areas where they can make real progress.

I tried out Pulumi for a couple weeks and they don't have a parameter or dynamic reference feature. They have a "config" system. https://www.pulumi.com/docs/reference/cli/pulumi_config/. It's very easy to use and has zero dependencies on external systems. I'm not saying that the cfn devs need to solve it the way they do, but they are definitely a data point for inspiration.

I want to be clear in that I'm not trying to dunk on cfn. I LOVE cfn. I would much prefer it over having to go outside the AWS ecosystem like you have to do with Terraform or Pulumi. Problems like this that go unaddressed make it hard to stick with it though.

ambsw-technology commented 4 years ago

I agree that we're talking about refining a not-ideal solution rather than clearly defining our problem(s) to seek something better, but we can't let perfect become the enemy of better. There are several problems converging on this one solution, we're probably not going to get multiple ideal solutions in a timely manner, and (as you said) a lot of us would get value from :latest.

I think the simplest possible solution should be proposed by the CF team (or just implemented), opening the door for feedback on additional improvements. I tried to propose a minimal but useful change in my post above:

Only support :latest in the top-level stack. Only update the parameter on change/deploy.

By constraining the solution to the top-level stack, there's no need to alter the behavior of nested stacks. Either the value is unchanged or it cascades. Specific versions can still be used in nested templates since changing the version necessarily updates the nested template.
Updating on change/deploy is the simplest approach and preserves existing expecatations.
It's possible that this would still need to pollute the parameter space (which is why the CF team should provide and detail a "simplest" proposal). If these are not parameters of the top-level template, I assume the template could be flagged as "unchanged" in a lot of situations. So it may be necessary to either support the syntax in the "default" field or add a specialized field.

mikerochip commented 4 years ago

Yep, I agree that being pragmatic and making the simplest change is a good call and I don't think that's mutually exclusive with rethinking the problem.

If we're going for simplicity then leaving off the version without additional rules is both the simplest thing and makes ssm params consistent with dynamic resolution of secrets.

I'll bring up the issues again in the community builders slack channel since there is a bunch of activity in devtools chats there, and because this ticket isn't the best platform for kicking off a long-form refactoring discussion.

arobthearab commented 4 years ago

+1 to implement a way to reference the latest value of an SSM parameter store parameter.

The impact on deterministic stack behavior is valid, but if too much weight is placed on such a consideration, it becomes a "best as the enemy of the good" situation. Make sure the potentially astonishing effects are documented (in general, at least). If you are concerned about the effects referencing a dynamic parameter version in CFN, don't do it. If you do it, be prepared for the potentially astonishing effects.

Of course, there is another way around this by changing SSM instead of CFN: allow a specific version of a parameter to be overwritten. Perhaps as a "don't version" attribute of the parameter.

There would certainly be an endless stream of "buts" with respect to that proposal as well, but there remains a problem to be solved: a way to share information dynamically between Lambda, CFN, SSM automations, and other API users/consumers.

robot-apocalypse commented 4 years ago

I will be "that guy" and say I strongly disagree that stack determinism should be a consideration here. So much of cloudformation already allows non-deterministic behavior (S3 references, lamda layers, etc.) I don't think it should suddenly be a concern here; to me that should be a separate feature request. Additionally, nothing we've been talking about would PREVENT deterministic stacks, it just wouldn't enforce it, right?

Is that discussion really why there's been no traction on this?

kirnberger1980 commented 4 years ago

Fully agree to what @wcurudy is pointing out here.

sannies commented 4 years ago

There is a decent workaround with custom resources. So: Don't let yourself be blocked! Do something about it!

  ConfigCustomResourceExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: [lambda.amazonaws.com]
            Action: ['sts:AssumeRole']
      Path: /
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: ['logs:*']
                Resource: 'arn:aws:logs:*:*:*'
              - Effect: Allow
                Action: ['ssm:GetParameter*']
                Resource: !Sub 'arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/*'

  ConfigCustomResourcePermissions:
    Type: AWS::Lambda::Permission
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: !GetAtt ConfigCustomResourceFunction.Arn
      Principal: 'cloudformation.amazonaws.com'

this is the permission infrastructure for the custom resource.

  ConfigCustomResourceFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: !Sub |
          import boto3
          import cfnresponse

          def handler(event, context):
            data = {}
            try:
                ssm = boto3.client('ssm')
                res = ssm.get_parameters_by_path(Path='/perhaps_a_prefix/', Recursive=True)

                for param in res['Parameters']:
                    data[param['Name'].replace('/perhaps_a_prefix/', '').replace('/', '.')] = param['Value']
                while 'NextToken' in res:
                    res = ssm.get_parameters_by_path(Path='/perhaps_a_prefix/, Recursive=True, NextToken=res['NextToken'])
                    for param in res['Parameters']:
                        data[param['Name'].replace('/perhaps_a_prefix/, '').replace('/', '.')] = param['Value']
                cfnresponse.send(event, context, cfnresponse.SUCCESS, data)
            except Exception as e:
                print(e)
                cfnresponse.send(event, context, cfnresponse.FAILED, {})

      Handler: index.handler
      Runtime: python3.6
      Role: !GetAtt ConfigCustomResourceExecutionRole.Arn

This lambda scrapes the completes config and makes it available in the custom resource. In the last step you define the custom resource.

  Config:
    Type: AWS::CloudFormation::CustomResource
    Properties:
      ServiceToken: !GetAtt ConfigCustomResourceFunction.Arn

if you got a config like /perhaps_a_prefix/server/prop_a and you need the latest value you can write it as:

...
 !Sub "${Config.server.prop_a}"
...

in my case /perhaps_a_prefix/ is the environment and it !Sub'ed into the code via ${env}. The '/' (slash) to '.' (dot) conversion is required as you can't have '/' in the !Sub expression.

ir-fuel commented 3 years ago

@sannies your code contains some typos ;)

izidorome commented 3 years ago

I think :latest or no number version at all is a simple and effective solution that would resolve most of our pain.

rhertogh commented 3 years ago

@mikerochip

I'm actually having a hard time understanding the utility of specifying the version number. Wouldn't the above use case be the usual reason for dynamically constructing the parameter name? Why is this dynamic behavior so different than the non-dynamic version, which doesn't force a version number?

It's most likely not an end user utility thing, but a cloudformation leaking it's implementation. My guess is that the template needs to resolve deterministically (for eg the changeset functionality), and there's not a straightforward way to resolve the latest parameter from a prior run to an actual version (without building a transform).

Yeah I'm pretty sure you hit the nail on the head here. Case in point: secrets DO let you leave off the version and use the latest, and they suffer from the exact problem you described.

I put in a feature request to force a resolve via a flag or capability or similar for secrets during change set creation: #369

To me it also looks like the deterministic nature and the necessity of defining a ChangeSet is the main challenge (@PatMyron could you please confirm this?). A possible solution would be to store the versions used of the dynamic values (at the time of creating the ChangeSet) in the Stack (similarly like the provided parameters are currently stored). Example of current parameters:

Now let's say, for example, we have a resource defined as follows (note the omission of the a version number):

MyRdsDbInstance:
    Type: AWS::RDS::DBInstance
    Properties:
        MasterUserPassword: '{{resolve:ssm-secure:MySecurePassword}}'

Would result in a "Dynamic Reference" record in the Stack in the following way:

In the example above the latest version of the MySecurePassword SSM parameter was 2 at the moment of creating the ChangeSet. When a new version of MySecurePassword is added a ChangeSet can pickup the version change when a Stack update is performed. This would allow for omitting the version number in the dynamic reference of ssm and ssm-secure parameters as well as fixing #369.

benkehoe commented 3 years ago

This seems like a good job for a Macro, which run before the change set is created. Maybe it could be rolled into the existing SecretsManager transform?

A Macro addresses both sides of this issue: it provides the mechanism to resolve the latest version into a version number, but it also results in a deployed stack using only version numbers. It also helps make it clear that the resolution happens at deployment time, and thus future updates to ssm or secrets manager values will require a new deployment.

aczire commented 3 years ago

Still waiting for "latest" or "no version". Really useful for lambda layer version update across stacks.

abjoerne commented 3 years ago

We are also missing the "latest" or "no version" or "0" as version-number. We rely on SSM for a lot of CF-templates for loosely coupled stacks. Also using CF in Service Catalog and as mentioned above by @Ricapar , SSM as parameter-type clutters the interface completely for users. Another option here would be to be able to hide the params from the user completely in the CF-wizard and Service Catalog.

We have a lot of default settings which are different in each account, and normally(!) are version 1 from the boot-strap-process, but in some accounts, but some times are in version 2 because of updates/changes initially. For example standardizes AZ to use for some services ("primary subnet"), standard db-retention in days, standard values for tags, standard prefixes etc.

If "latest" have cases where it will not work well or are breaking your setup, then do not use it. I see no suggestion here to remove the ability to use versioning, so those needing it or relying on it can continue doing so?!

jncheung commented 3 years ago

This feature is definitely a needed one as the CDK is already supporting this feature when the version is omitted.

https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ssm.StringParameterAttributes.html#version

Since cloudformation and CDK both maintain by AWS and it really should not have this mis-alignment.

Hoping to see the same feature available in CloudFormation very soon.

achdevops commented 3 years ago

meanwhile, we could use this workaround:

define the parameter section with default value

MyPort: Type: AWS::SSM::Parameter::Value Default: '/env/default/value

then reference in the resource

**SecurityGroupIngress:

IpProtocol: tcp FromPort: !Ref MyPort ToPort: !Ref MyPort CidrIp: 0.0.0.0/0**

we hope that it help!

robot-apocalypse commented 3 years ago

Unfortunately that doesn't really solve the problem, as you still can't dynamically construct the parameter name ala /app/env-name/db/dbname.

tyliggity commented 3 years ago

@wcurudy Actually, you definitely can and in fact it's likely the best workaround:

# ...
  Fn::Join:
    - ''
    - - '{{resolve:ssm:'
      - !Sub '${Environment}-api-vpce-dns'
      - !Sub ':${PrivateApiVpceDnsParamVersion}}}'
# ...

In my case, the name of the param is dynamically set by a template parameter representing the environment name. The version of the param is dynamically set by a template parameter too. I have the current latest version set as the default parameter value as it won't change much at all. However, if it does change often in your case you can always execute the template with parameter overrides.

Happy templating!

robot-apocalypse commented 3 years ago

@tyliggity Correct me if I'm wrong, but that can't be done in the "default" section of a parameter, which is the workaround suggested by achdevops to which I was replying.

tyliggity commented 3 years ago

Honestly, I haven't tried. I don't see why it wouldn't work there though.

robot-apocalypse commented 3 years ago

According to this: https://stackoverflow.com/questions/62395211/default-value-aws-replace-with-pseudo-parameter-reference, it does not. This also jives with my memory of running into this issue two years ago, though I don't have time to test it again right now.

The point of this feature request is to be able to dynamically construct SSM parameter names (i.e. use "dynamic" SSM references) without having to lock it to a particular version, identical to how secrets manager dynamic references work. Just specifying the version as a parameter doesn't change that, you've just moved it from the template to a parameter file. Neither of these workarounds solve both of those problems. Until then, SSM parameter store can never be a complete parameter store solution. You will always need to specify at least some parameters (e.g. parameter versions) somewhere else.

ajlozier commented 3 years ago

There is no good argument for not implementing this behavior, which is consistent with Secrets manager. And not implementing it is making things more difficult. Personally I ended up moving my parameters to the application level, which is able to fetch the most recent version just fine. However I know this does not fit everyone’s use case.

robot-apocalypse commented 3 years ago

Not only is it not consistent with secrets manager, it is not consistent with other mechanisms for retrieving values from SSM Parameter Store. The API does not require a version, neither does the console nor the AWS::SSM::Parameter::Name CF parameter type. It is quite clearly a bug, in design if not in code, and I doubt it will get addressed as long as it is classified as a "feature request". I think it would be more appropriate to reopen 716, and close this ticket.

@ajlozier I've also been recommending my clients manage parameters at the application level; in a lot of situations this is actually preferable anyway, since they can be resolved at runtime that way. The problem of course are parameters that need to be used within the template, like an instance size being passed to an EC2 resource. I also don't like that the values propagate to all uses immediately, rather than being able to control the distribution via stack updates, but I supposed that's a little nit-picky

As far as I can tell, no one from AWS has weighed in on feasibility or when/how/if this will ever be fixed, or why it should not. Nothing has changed in the ticket in the 2 years I've been watching it, no milestone, no assignee, nothing. We are on our own. So here is a summary of the "workarounds" I know of, and their associated shortcomings.

Specify parameter names in "default" config of cf parameters. This defaults to "latest" like you'd expect, but there's no way to dynamically construct the name
Construct the SSM parameter name in-template, using dynamic SSM references. Requires you to specify the version number for some reason
Write a macro to retrieve the latest version number and preprocess the template. Breaks other features like nested stacks: 664, and maybe change sets? See 716
Handle retrieving parameters at the application level. Not valid for in-template use of parameter values, and updates propagate immediately rather than as stacks update

Let me know if I missed any.

tyliggity commented 3 years ago

I don't see how it's not a workaround considering parameters can be specified every time the template is executed and that's how most template pipelines are ran. It's like how are you tracking your Docker container version? While Docker supports "latest" it's certainly not best practice nor is it used very much in the real world for that reason. You have to manage how that's done.

So if you have to track versions around your solution anyway, these SSM params are no different. The pain was that the version had to be hard coded but as I showed it actually does not, it can be dynamic. If the complaint is that we have to track versions at all then that's just not valid considering software best practices as we see in other areas of the stack.

El lun., 5 de abr. de 2021 11:07, wcurudy @.***> escribió:

According to this: https://stackoverflow.com/questions/62395211/default-value-aws-replace-with-pseudo-parameter-reference, it does not. This also jives with my memory of running into this issue two years ago, though I don't have time to test it again right now.

The point of this feature request is to be able to dynamically construct SSM parameter names (i.e. use "dynamic" SSM references) without having to lock it to a particular version, identical to how secrets manager dynamic references work. Just specifying the version as a parameter doesn't change that, you've just moved it from the template to a parameter file. Neither of these workarounds solve both of those problems. Until then, SSM parameter store can never be a complete parameter store solution. You will always need to specify at least some parameters (e.g. parameter versions) somewhere else.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/75#issuecomment-813474958, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC4RO4ISJTSTUZJLOMN5CMTTHHN4BANCNFSM4IIQT7TA .

robot-apocalypse commented 3 years ago

It absolutely is best practice to use docker :latest in MANY MANY situations, most obviously development environments. Just this morning I used :latest to test something on ubuntu; why would I need to lock it down to a specific release, i just needed ubuntu. So docker helpfully gives me the option to just grab the latest. Saying that's not "best practice" is pretty narrow minded. Stable production release isn't the only use case we're considering, we're talking the entire development lifecycle, including developers just screwing around.
Comparing parameter versions to container versions is just...weird. apples to oranges. SSM Secrets is a much better comparison.
If you need to change 25 out of 800 parameters used by 100 templates in 25 different environments and 3 regions, where do you store the parameter version numbers that are currently "valid"? SSM again? Another parameter store? A file? Why not just put the whole parameter in the file since you have to maintain it anyway? if only there was a shorthand way to indicate on the parameter ITSELF what the "valid" version was....
REAL best practice would be to version your parameters in SCM, load them into SSM when they need to be updated, and reference them in your templates. Only in your production environment would you lock it down to a specific version, mostly as a safety precaution.

Your workaround isn't a complete workaround because it doesn't solve the problems as stated in the feature request. That's just how words work. What you're actually saying is you don't think it's that big of a deal, which is fine for you and your narrow use case, but I and many others on here disagree.

ambsw-technology commented 3 years ago

If you need to change 25 out of 800 parameters used by 100 templates in 25 different environments and 3 regions, where do you store the parameter version numbers that are currently "valid"? SSM again? Another parameter store? A file? Why not just put the whole parameter in the file since you have to maintain it anyway? if only there was a shorthand way to indicate on the parameter ITSELF what the "valid" version was....

I keep coming back to some flavor of this. As I mentioned a year ago, the only place where it seems like an advantage to store a version number rather than a raw value is for a secure values.

... and as far as I can tell, the only reason not to provide this functionality more than a year ago is a deliberate decision to try and make the "free" SSM feature less useful than the paid Secrets.

tyliggity commented 3 years ago

Just because using latest is convenient does not mean it is best practice. It's not.

Almost any reputable IT operation would never use "latest" Docker tags in their production images. Such a practice would make repeatable builds impossible. How does laziness justify the drawbacks of using "latest"?

And since cloudformation is IaC it would also be best practice to push a change when the parameters version should be changed.

Convenience is rarely best practice. If you have an 800 parameter use case, using latest is even MORE risky and definitely not best practice.

I stand by the workaround. The feature requested here would be convenient, that's all.

On Tue, Apr 6, 2021 at 11:44 AM wcurudy @.***> wrote:

It absolutely is best practice to use docker :latest in MANY MANY situations, most obviously development environments. Just this morning I used :latest to test something on ubuntu; why would I need to lock it down to a specific release, i just needed ubuntu. So docker helpfully gives me the option to just grab the latest. Saying that's not "best practice" is pretty narrow minded. Stable production release isn't the only use case we're considering, we're talking the entire development lifecycle, including developers just screwing around.

Comparing parameter versions to container versions is just...weird. apples to oranges. SSM Secrets is a much better comparison.

If you need to change 25 out of 800 parameters used by 100 templates in 25 different environments and 3 regions, where do you store the parameter version numbers that are currently "valid"? SSM again? Another parameter store? A file? Why not just put the whole parameter in the file since you have to maintain it anyway? if only there was a shorthand way to indicate on the parameter ITSELF what the "valid" version was....

REAL best practice would be to version your parameters in SCM, load them into SSM when they need to be updated, and reference them in your templates. Only in your production environment would you lock it down to a specific version, mostly as a safety precaution.

Your workaround isn't a complete workaround because it doesn't solve the problems as stated in the feature request. That's just how words work. What you're actually saying is you don't think it's that big of a deal, which is fine for you and your narrow use case, but I and many others on here disagree.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/75#issuecomment-814224992, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC4RO4O4U5KUPJNLT2FXLJTTHMT4PANCNFSM4IIQT7TA .

jplock commented 3 years ago

Specifying the version is now optional on https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-ssm

aws-cloudformation / cloudformation-coverage-roadmap