Closed simbleau closed 10 months ago
You mention ubuntu-latest
in CI, so this is in GitHub Actions? The error you're getting is from the IMDS credentials provider, which is the very last one to be tried. That means the environment credentials provider already failed to find credentials in those env vars by the time it got to that error.
You can set up the tracing-subscriber
crate with the env-filter
feature, and initialize it with tracing_subscriber::fmt::init()
in main. Doing so will allow you to run your program with RUST_LOG=debug
to see what's going on.
I suspect the environment variables aren't actually getting set in the GitHub Actions workflow config.
You mention
ubuntu-latest
in CI, so this is in GitHub Actions? The error you're getting is from the IMDS credentials provider, which is the very last one to be tried. That means the environment credentials provider already failed to find credentials in those env vars by the time it got to that error.You can set up the
tracing-subscriber
crate with theenv-filter
feature, and initialize it withtracing_subscriber::fmt::init()
in main. Doing so will allow you to run your program withRUST_LOG=debug
to see what's going on.I suspect the environment variables aren't actually getting set in the GitHub Actions workflow config.
Yes! it is GitHub actions.
Let me investigate further and report back. I think you're probably right.
Logs
2023-12-08T03:09:48.277464Z DEBUG aws_config::fs_util: loaded home directory src="HOME"
2023-12-08T03:09:48.277601Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/home/runner" path="~/.aws/config"
2023-12-08T03:09:48.277655Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/home/runner/.aws/config"
2023-12-08T03:09:48.277696Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file not found path=~/.aws/config
2023-12-08T03:09:48.277719Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/home/runner/.aws/config") size=0
2023-12-08T03:09:48.277790Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/home/runner" path="~/.aws/credentials"
2023-12-08T03:09:48.277807Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/home/runner/.aws/credentials"
2023-12-08T03:09:48.277825Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2023-12-08T03:09:48.277836Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/home/runner/.aws/credentials") size=0
2023-12-08T03:09:48.289494Z DEBUG build_profile_provider: hyper_rustls::config: with_native_roots processed 137 valid and 0 invalid certs
2023-12-08T03:09:48.342353Z DEBUG invoke{service=dynamodb operation=Scan}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'serialization' phase
2023-12-08T03:09:48.342537Z DEBUG invoke{service=dynamodb operation=Scan}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'before transmit' phase
2023-12-08T03:09:48.342633Z DEBUG invoke{service=dynamodb operation=Scan}:try_op: aws_smithy_runtime::client::retries::strategy::standard: no client rate limiter configured, so no token is required for the initial request.
2023-12-08T03:09:48.342652Z DEBUG invoke{service=dynamodb operation=Scan}:try_op: aws_smithy_runtime::client::orchestrator: retry strategy has OKed initial request
2023-12-08T03:09:48.342680Z DEBUG invoke{service=dynamodb operation=Scan}:try_op: aws_smithy_runtime::client::orchestrator: beginning attempt #1
2023-12-08T03:09:48.342743Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator::endpoints: resolving endpoint endpoint_params=EndpointResolverParams(TypeErasedBox[!Clone]:Params { region: Some("us-east-1"), use_dual_stack: false, use_fips: false, endpoint: None }) endpoint_prefix=None
2023-12-08T03:09:48.342800Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator::endpoints: will use endpoint Endpoint { url: "https://dynamodb.us-east-1.amazonaws.com/", headers: {}, properties: {} }
2023-12-08T03:09:48.343113Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=the credential provider was not enabled: environment variable not set (CredentialsNotLoaded(CredentialsNotLoaded { source: "environment variable not set" }))
2023-12-08T03:09:48.343187Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Profile context=the credential provider was not enabled: No profiles were defined (CredentialsNotLoaded(CredentialsNotLoaded { source: NoProfilesDefined }))
2023-12-08T03:09:48.343234Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=WebIdentityToken context=the credential provider was not enabled: $AWS_WEB_IDENTITY_TOKEN_FILE was not set (CredentialsNotLoaded(CredentialsNotLoaded { source: "$AWS_WEB_IDENTITY_TOKEN_FILE was not set" }))
2023-12-08T03:09:48.343291Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=EcsContainer context=the credential provider was not enabled: ECS provider not configured (CredentialsNotLoaded(CredentialsNotLoaded { source: "ECS provider not configured" }))
2023-12-08T03:09:48.343328Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loading credentials from IMDS
2023-12-08T03:09:48.343519Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'serialization' phase
2023-12-08T03:09:48.343600Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'before transmit' phase
2023-12-08T03:09:48.343643Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op: aws_smithy_runtime::client::retries::strategy::standard: no client rate limiter configured, so no token is required for the initial request.
2023-12-08T03:09:48.343669Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op: aws_smithy_runtime::client::orchestrator: retry strategy has OKed initial request
2023-12-08T03:09:48.343702Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op: aws_smithy_runtime::client::orchestrator: beginning attempt #1
2023-12-08T03:09:48.343752Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator::endpoints: resolving endpoint endpoint_params=EndpointResolverParams(TypeErasedBox[!Clone]:()) endpoint_prefix=None
2023-12-08T03:09:48.343808Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator::endpoints: will use endpoint Endpoint { url: "http://169.254.169.254/", headers: {}, properties: {} }
2023-12-08T03:09:48.343925Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt: aws_config::imds::client::token: IMDS token cache miss
2023-12-08T03:09:48.344127Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'serialization' phase
2023-12-08T03:09:48.344238Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'before transmit' phase
2023-12-08T03:09:48.344286Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op: aws_smithy_runtime::client::retries::strategy::standard: no client rate limiter configured, so no token is required for the initial request.
2023-12-08T03:09:48.344315Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op: aws_smithy_runtime::client::orchestrator: retry strategy has OKed initial request
2023-12-08T03:09:48.344353Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op: aws_smithy_runtime::client::orchestrator: beginning attempt #1
2023-12-08T03:09:48.344406Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator::endpoints: resolving endpoint endpoint_params=EndpointResolverParams(TypeErasedBox[!Clone]:()) endpoint_prefix=None
2023-12-08T03:09:48.344464Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator::endpoints: will use endpoint Endpoint { url: "http://169.254.169.254/", headers: {}, properties: {} }
2023-12-08T03:09:48.344592Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'transmit' phase
2023-12-08T03:09:48.344829Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op:try_attempt: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-12-08T03:09:48.345849Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op:try_attempt: hyper::client::connect::http: connected to 169.254.169.254:80
2023-12-08T03:09:48.346255Z DEBUG hyper::proto::h1::io: flushed 255 bytes
2023-12-08T03:09:48.346559Z DEBUG hyper::proto::h1::io: parsed 5 headers
2023-12-08T03:09:48.346573Z DEBUG hyper::proto::h1::conn: incoming body is content-length (344 bytes)
2023-12-08T03:09:48.346604Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-12-08T03:09:48.346777Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'before deserialization' phase
2023-12-08T03:09:48.346841Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'deserialization' phase
2023-12-08T03:09:48.346993Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'after deserialization' phase
2023-12-08T03:09:48.347090Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op: aws_smithy_runtime::client::retries::strategy::standard: encountered unretryable error attempts=1 max_attempts=4
2023-12-08T03:09:48.347126Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt:invoke{service=imds operation=get-token}:try_op: aws_smithy_runtime::client::orchestrator: a retry is either unnecessary or not possible, exiting attempt loop
2023-12-08T03:09:48.347231Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator: encountered orchestrator error; halting
2023-12-08T03:09:48.347303Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op: aws_smithy_runtime::client::retries::strategy::standard: encountered unretryable error attempts=1 max_attempts=4
2023-12-08T03:09:48.347332Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:invoke{service=imds operation=get}:try_op: aws_smithy_runtime::client::orchestrator: a retry is either unnecessary or not possible, exiting attempt loop
2023-12-08T03:09:48.347428Z WARN invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider failed to provide credentials provider=Ec2InstanceMetadata error=an error occurred while loading credentials: failed to load IMDS session token: service error: invalid token (ProviderError(ProviderError { source: FailedToLoadToken(FailedToLoadToken { source: ServiceError(ServiceError { source: TokenError { kind: InvalidToken }, raw: Response { status: StatusCode(411), headers: Headers { headers: {"content-type": HeaderValue { _private: "text/html; charset=us-ascii" }, "server": HeaderValue { _private: "Microsoft-HTTPAPI/2.0" }, "date": HeaderValue { _private: "Fri, 08 Dec 2023 03:09:47 GMT" }, "connection": HeaderValue { _private: "close" }, "content-length": HeaderValue { _private: "344" }} }, body: SdkBody { inner: Once(Some(b"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\"[http://www.w3.org/TR/html4/strict.dtd\](http://www.w3.org/TR/html4/strict.dtd/)">\r\n<HTML><HEAD><TITLE>Length Required</TITLE>\r\n<META HTTP-EQUIV=\"Content-Type\" Content=\"text/html; charset=us-ascii\"></HEAD>\r\n<BODY><h2>Length Required</h2>\r\n<hr><p>HTTP Error 411. The request must be chunked or have a content length.</p>\r\n</BODY></HTML>\r\n")), retryable: true }, extensions: Extensions } }) }) }))
@jdisanti - Appears as though due to not seeing the environment variables,
2023-12-08T03:09:48.360329Z DEBUG invoke{service=dynamodb operation=Scan}:try_op:try_attempt:lazy_load_identity:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=the credential provider was not enabled: environment variable not set (CredentialsNotLoaded(CredentialsNotLoaded { source: "environment variable not set" }))
However this is false... The environment variables are definitely there and correct.
Secrets in GitHub Actions aren't environment variables. There's a little extra piping you have to do in the workflow config, for example: https://github.com/smithy-lang/smithy-rs/blob/main/.github/workflows/ci.yml#L34-L35
Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.
Perfect! Thanks
Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
Describe the bug
When I run locally, with a
~/.aws/credentials
file, everything works fine! When this runs on ubuntu-latest CI using environment variablesAWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
, I get a very confusing error.My hunch is that this is a symptom of a different issue.
The error
How I'm using it
Expected Behavior
N/A
Current Behavior
N/A
Reproduction Steps
N/A
Possible Solution
No response
Additional Information/Context
No response
Version
Environment details (OS name and version, etc.)
MacOS Ventura (my machine), ubuntu (CI machine)
Logs
No response