Open dogfish182 opened 1 year ago
To put a bit more context on this issue, I'm confused by the datadog instructions on how to setup this integration (and have a support ticket running).
This page https://github.com/DataDog/cloudformation-template/tree/master/aws
^^ says it will setup datadog for you, however one of the first steps is to manually provision your accounts in datadog and copy the externalID as a parameter before you manually run cloudformation. (not really doable at any kind of scale). At the end of the doc it says you can use THIS integration if you wish to manage the integration, this seems like circular logic, because if I already set it up manually then it's unmanaged now?
What I would like to achieve is to use this integration which creates the datadog side resources and then create the AWS side resource myself and input the externalID into the role I'm creating, by reading the secrets manger entry that this extension writes.
Has anyone been able to achieve this?
Thanks for your contribution!
This issue has been automatically marked as stale because it has not had activity in the last 30 days. Note that the issue will not be automatically closed, but this notification will remind us to investigate why there's been inactivity. Thank you for participating in the Datadog open source community.
If you would like this issue to remain open:
Verify that you can still reproduce the issue in the latest version of this project.
Comment that the issue is still reproducible and include updated details requested in the issue template.
I can still reproduce this issue as shown in the orginal post.
We are facing the same error (although on Monitor and Dashboards) We have a support case open with AWS
We are facing the same error (although on Monitor and Dashboards) We have a support case open with AWS
I did the same and they told us we need to contact datadog as the error is being swallowed by the custom cloudformation resource handler.
Thanks we just did the same, lets see what happens
Thanks for opening this issue. We are going to merge and release the change https://github.com/DataDog/datadog-cloudformation-resources/pull/258 which should catch any unhandled exceptions in the resources them selves.
However, this wouldn't expose all errors mainly because AWS does obfuscate logs/events quite heavily on their end so things such as bad type configuration and bad execution roles would still fail in non-obvious ways. Which I suspect is the reason for the failures you are seeing @flavioelawi with dashboards and monitors
We have resolved our issue;
our execution role already had the correct trust policy:
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"resources.cloudformation.amazonaws.com",
"cloudformation.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
And a policy to allow access to the Secrets and its Kms key
{
"Action": [
"secretsmanager:GetSecretValue",
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*",
"Effect": "Allow"
},
We also added the CloudWatchLogsFullAccess
managed policy to allow for the integration to push logs to Cloudwatch logs (but its log group is still empty, I guess for another issue)
The issue in our case was a typo in the dynamic reference, where we were missing the SecretString
part before the Json attribute selector.
@dogfish182 in your case you are missing external_id
from your dynamic reference at the end, this is what is setup by the integration lambda/code
Also some feedback:
We released the AWS resource version 2.4.0 that should capture and return any unhandled exception on the resource it self. However, as mentioned previously, errors swallowed by AWS would probably still not be captured by this change as it happens outside of the resource handler.
Describe the bug AWS integration fails with obscure error
To Reproduce Steps to reproduce the behavior: run a template that looks like this
Logs
Expected behavior The cloudformation should run to completion. I expect the account integration to enable the account in datadog (this does occur) I expect the secret to be written to secrets manager (this does NOT occur) I expect my role to be created which I pull the secret from secrets manager (this does NOT occur)
Environment and Versions (please complete the following information): Datadog AWS Integration 2.2.1 I am generating cloudformation via cdkv2 however I doubt this is relevant as I've included the generated cloudformation template above (which is run and faults).
Additional context It essentially looks like the cloudformation handler is swallowing the error, which makes it very hard to troubleshoot this. I've also logged a ticket with datadog support.