Open Yethal opened 3 months ago
Hi @Yethal! We've got a tutorial for GCP up already https://developer.hashicorp.com/nomad/tutorials/fed-workload-identity/integration-gcp And this blog post has an example for GitHub Actions: https://www.hashicorp.com/blog/nomad-jwt-auth-with-github-actions
But we definitely want to have tutorials for AWS and Azure as well, they're just going to take a while to get done.
Hey @tgross. Thanks for the links, unfortunately we use AWS but I'll go through the GCP guide, perhaps the setup is similar enough that this will be sufficient.
For whatever its worth, @Yethal, -- and you're probably already aware of this -- there is an example video here showing a very brief example of using Workload Identity with AWS (which is nice but it does leave much to be desired).
That said, I would also love to see Hashicorp publish some proper documentation on how users can leverage Nomad's Workload Identity with AWS IAM.
(For internal Hashi folks; see: Support Request #150900)
While we're working on a proper doc, here's an excerpt from my notes describing the configuration flow for AWS:
the Nomad OIDC endpoints (.well-known/openid-configuration
and .well-known/jwks.json
) need to be publicly reachable over HTTPS on port 443 to be verified. Vault is an exception when [[#JWKS public keys]] are used.
The Nomad servers' configuration file needs to contain the oidc_issuer
parameter with the FQDN (including https://
) on which the Nomad Servers are available:
oidc_issuer = "https://nomad.company.com/"
In AWS, an OIDC Identity Provider needs to be created in IAM. When creating it via Terraform, the API or CLI, the SHA1 thumbprint of the TLS certificate protecting the publicly available FQDN needs to be manually obtained. Via the web console it's automatic.
When using Terraform to configure the Identity Provider, the TLS provider can be used to automatically fetch all relevant thumbprints:
data "tls_certificate" "cert" {
url = "$NOMAD_ADDR"
}
resource "aws_iam_openid_connect_provider" "nomad" {
url = data.tls_certificate.cert.url
thumbprint_list = data.tls_certificate.cert.certificates[*].sha1_fingerprint
}
Once the Identity Provider is created, AWS IAM roles which allow Nomad workloads to assume them, and give access to AWS resources, can be created. Example AWS IAM Role trust policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/$OIDC_PROVIDER"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"$OIDC_PROVIDER:aud": "aws"
}
}
}
]
}
In this case, the trust policy allows all Nomad workloads with the audience (aud
) of aws
to assume the role. This ensures that only identities meant for AWS auth are allowed. Any other field from the JWT, such as the nomad_namespace
, nomad_task
or nomad_job_id
can be used to further filter down exactly which workload can assume the IAM role.
To provision a JWT identity token that can use the above trust relationship, an identity
block is needed on the task level in the Nomad job file:
identity {
name = "aws"
aud = ["aws"] #audience needs to match with the Trust Policy and provider configuration
file = true #file so that it is stored in a file used by AWS CLI/SDKs
env = true
ttl = "1h" #JWT TTL, not related to AWS' 1h default assume role TTL
}
as well as configuration instructing the AWS CLI or SDK of what role it can assume:
env {
AWS_ROLE_ARN = "arn:aws:iam::$AWS_ACCOUNT_ID:role/$ROLE_THAT_CAN_BE_ASSUMED_BY_THIS_IDENTITY"
AWS_WEB_IDENTITY_TOKEN_FILE = "${NOMAD_SECRETS_DIR}/nomad_aws.jwt" #the name format is nomad_$NAME_OF_IDENTITY.jwt
}
The AWS SDKs will automatically reauthenticate when their session expires using the latest available JWT, so its TTL is somewhat irrelevant.
I've documented how to configure Entra ID as OIDC in our company blog some months ago. Just in case anyone needs that https://blog.bitexpert.de/blog/nomad_oidc_entra_id
Hey @Yethal!! We've published a HashiCorp Knowledge Base Article on this topic. You can find it Integrating AWS IAM with HashiCorp Nomad Workload Identity.
@SuyashHashiCorp Thank you so much for this
Proposal
Nomad 1.7 added ability to integrate workload identities with external IDPs. It would be super useful if Nomad docs contained setup guides for popular cloud providers such as AWS, Azure etc. For comparison, here's similar guide for setting up Github Actions: https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services
Use-cases
Using OIDC instead of hardcoded aws credentials in jobs
Attempted Solutions
I really don't want to figure out security settings via trial-and-error.