search

hairyhenderson / gomplate

A flexible commandline tool for template rendering. Supports lots of local and remote datasources.
https://gomplate.ca
MIT License
2.73k stars 187 forks source link

Secrets Manager Datasource Doesn't Appear to Work #2190

Open sjiveson opened 4 months ago

sjiveson commented 4 months ago 
./gomplate --version
gomplate version 4.1.0

aws --version
aws-cli/1.24.10 Python/3.6.8 Linux/4.18.0-553.8.1.el8_10.x86_64 botocore/1.26.10

cat /etc/redhat-release
Red Hat Enterprise Linux release 8.10 (Ootpa)

Hi,

I'm attempting to retrieve secrets using AWS Secrets Manager (on an EC2 instance), however, I'm getting this error:

ec2-user]# ./gomplate --verbose -d 'foo=aws+sm:' -i '{{ ds "foo" "goemplate" }}'
13:32:18 DBG starting gomplate
13:32:18 DBG config is:
---
datasources:
  foo:
    header: {}
    url: 'aws+sm:'
in: '{{ ds "f...'
 version=4.1.0 build=cc2584028866967a39b096265d5b9af4516c734f
13:32:18 DBG completed rendering templatesRendered=0 errors=1 duration=84.088165ms
13:32:18 ERR  err="renderTemplate: failed to render template <arg>: template: <arg>:1:3: executing \"<arg>\" at <ds \"foo\" \"goemplate\">: error calling ds: couldn't read datasource 'foo' (aws+sm:///goemplate): stat (url: \"aws+sm:///\", name: \"goemplate\"): stat goemplate: file does not exist"

Looking at my CloudTrail logs, I'm seeing this:

...
errorCode   ResourceNotFoundException
errorMessage    Secrets Manager can't find the specified secret.
eventCategory   Management
eventID NNee72f2-f237-4467-81b4-fe0d81f8cNNN
eventName   GetSecretValue
eventSource secretsmanager.amazonaws.com
eventTime   2024-07-23T13:23:37Z
eventType   AwsApiCall
eventVersion    1.09
managementEvent 1
readOnly    1
requestParameters.secretId  /goemplate
...

As you can see on that last line, a forward slash is being included at the start of the secretId when it shouldn't be. This also occurs if I specify, for instance, goemplate/something or goemplate/something/something.

Any assistance appreciated and thanks for a great tool.

hairyhenderson commented 4 months ago

I think this may a duplicate to #2180 - as a workaround, try setting the region?

sjiveson commented 4 months ago

It's not that unfortunately, the error message when I hadn't was very clear which was nice:

15:46:40 ERR  err="renderTemplate: failed to render template one.txt.tmpl: template: one.txt.tmpl:1:32: executing \"one.txt.tmpl\" at <ds \"testsource\" \"/goemplate/awssm/test\">: error calling ds: couldn't read datasource 'testsource' (aws+sm:///goemplate/awssm/test): stat (url: \"aws+sm:///\", name: \"goemplate/awssm/test\"): stat test: getSecretValue: operation error Secrets Manager: GetSecretValue, failed to resolve service endpoint, endpoint rule error, Invalid Configuration: Missing Region"
hairyhenderson commented 4 months ago

Ah - this reminds me of some strangeness I had to deal with in the old v3 implementation. Looks like some of that got lost in the v4/go-fsimple shuffle.

github-actions[bot] commented 2 months ago

This issue is stale because it has been open for 60 days with no activity. If it is no longer relevant or necessary, please close it. Given no action, it will be closed in 14 days.

If it's still relevant, one of the following will remove the stale marking:

sjiveson commented 2 months ago

I'm not capable but a fix would still be much appreciated

github-actions[bot] commented 1 week ago

This issue is stale because it has been open for 60 days with no activity. If it is no longer relevant or necessary, please close it. Given no action, it will be closed in 14 days.

If it's still relevant, one of the following will remove the stale marking:

sjiveson commented 1 week ago

And again. I'm not capable but a fix would still be much appreciated

hairyhenderson commented 3 days ago

@sjiveson I'm sorry I haven't been able to solve this yet. My time available to work on gomplate is limited. I'll try to find some time over the next few days.

hairyhenderson commented 3 days ago

@sjiveson I've actually started digging into this a bit - just to double-check - goemplate isn't a typo right?

And, have you tried referring to the key directly?

$ gomplate --verbose -d 'foo=aws+sm:goemplate' -i '{{ ds "foo" }}'

It's somewhat unusual to specify the whole aws+sm scheme as the datasource, though should not be invalid.

I think what's going on is the URL parsing has no special-case for the aws+sm scheme anymore - the non-prefixed argument is being interpreted as a regular path rooted in /, whereas with aws+sm the old behaviour was to build an opaque URL from that (with no leading /).

sjiveson commented 1 day ago

Hey. Thank you for responding - I know the feeling well, hence my own delay in responding 😏 .

It was a typo with the secret name but I didn't bother changing it in Secrets Manager and it is the correct name of the secret.

Referring to the key directly as you suggested - v3:

AWS_REGION="eu-west-2" ./gomplatev3 --verbose -d 'foo=aws+sm:goemplate' -i '{{ ds "foo" }}'
17:22:10 DBG starting gomplate
17:22:10 DBG config is:
---
datasources:
  foo:
    header: {}
    url: aws+sm:goemplate
in: '{{ ds "f...'
 build=f525a6ea version=3.11.8
SECRET-VALUE-HERE17:22:11 DBG completed rendering duration=0.119866419 errors=0 templatesRendered=1

With v4:

AWS_REGION="eu-west-2" ./gomplatev4 --verbose -d 'foo=aws+sm:goemplate' -i '{{ ds "foo" }}'
17:22:19 DBG starting gomplate
17:22:19 DBG config is:
---
datasources:
  foo:
    header: {}
    url: aws+sm:goemplate
in: '{{ ds "f...'
 version=4.1.0 build=cc2584028866967a39b096265d5b9af4516c734f
17:22:19 DBG completed rendering templatesRendered=0 errors=1 duration=174.397089ms
17:22:19 ERR  err="renderTemplate: failed to render template <arg>: template: <arg>:1:3: executing \"<arg>\" at <ds \"foo\">: error calling ds: couldn't read datasource 'foo' (aws+sm:goemplate): stat (url: \"aws+sm:///\", name: \"goemplate\"): stat goemplate: file does not exist"

I should be able to respond more quickly now I'm setup to test again.