benesch
commented
1 year ago Eyeballing the output of some unaffected clusterd processes by hand, though, I'm really not seeing that debugging println on any of those unaffected processes. We see STS errors around 4pm ET (as expected) that continue for a bit and then clear right up:
clusterd {"timestamp":"2023-06-13T20:09:02.084937Z","level":"INFO","fields":{"message":"credentials cache returned CredentialsNotLoaded, ignoring"},"target":"aws_http::auth","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:02.774907Z","level":"INFO","fields":{"message":"credentials cache returned CredentialsNotLoaded, ignoring"},"target":"aws_http::auth","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:02.802298Z","level":"WARN","fields":{"message":"STS returned an error assuming web identity role","error":"service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:02 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }))"},"target":"aws_config::web_identity_token","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:02.802611Z","level":"WARN","fields":{"message":"provider failed to provide credentials","provider":"WebIdentityToken","error":"an error occurred while loading credentials: service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ProviderError(ProviderError { source: ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:02 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }) }))"},"target":"aws_config::meta::credentials::chain","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:02.815222Z","level":"INFO","fields":{"message":"credentials cache returned CredentialsNotLoaded, ignoring"},"target":"aws_http::auth","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:03.383098Z","level":"INFO","fields":{"message":"credentials cache returned CredentialsNotLoaded, ignoring"},"target":"aws_http::auth","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.116729Z","level":"WARN","fields":{"message":"STS returned an error assuming web identity role","error":"service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:03 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }))"},"target":"aws_config::web_identity_token","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.117057Z","level":"WARN","fields":{"message":"provider failed to provide credentials","provider":"WebIdentityToken","error":"an error occurred while loading credentials: service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ProviderError(ProviderError { source: ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:03 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }) }))"},"target":"aws_config::meta::credentials::chain","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.138912Z","level":"INFO","fields":{"message":"credentials cache returned CredentialsNotLoaded, ignoring"},"target":"aws_http::auth","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.228762Z","level":"WARN","fields":{"message":"STS returned an error assuming web identity role","error":"service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:04 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }))"},"target":"aws_config::web_identity_token","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.229078Z","level":"WARN","fields":{"message":"provider failed to provide credentials","provider":"WebIdentityToken","error":"an error occurred while loading credentials: service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ProviderError(ProviderError { source: ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:04 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }) }))"},"target":"aws_config::meta::credentials::chain","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.229290Z","level":"INFO","fields":{"message":"external operation fetch_batch::get failed, retrying in 32ms: indeterminate: s3 get meta err: failed to construct request"},"target":"mz_persist_client::internal::machine"}
clusterd {"timestamp":"2023-06-13T20:09:04.266399Z","level":"INFO","fields":{"message":"credentials cache returned CredentialsNotLoaded, ignoring"},"target":"aws_http::auth","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.290968Z","level":"WARN","fields":{"message":"STS returned an error assuming web identity role","error":"service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:03 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }))"},"target":"aws_config::web_identity_token","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.291272Z","level":"WARN","fields":{"message":"provider failed to provide credentials","provider":"WebIdentityToken","error":"an error occurred while loading credentials: service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ProviderError(ProviderError { source: ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:03 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }) }))"},"target":"aws_config::meta::credentials::chain","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.291716Z","level":"INFO","fields":{"message":"external operation fetch_batch::get failed, retrying in 64ms: indeterminate: s3 get meta err: failed to construct request"},"target":"mz_persist_client::internal::machine"}
clusterd {"timestamp":"2023-06-13T20:09:04.364927Z","level":"INFO","fields":{"message":"credentials cache returned CredentialsNotLoaded, ignoring"},"target":"aws_http::auth","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.394375Z","level":"WARN","fields":{"message":"STS returned an error assuming web identity role","error":"service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:03 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }))"},"target":"aws_config::web_identity_token","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.394685Z","level":"WARN","fields":{"message":"provider failed to provide credentials","provider":"WebIdentityToken","error":"an error occurred while loading credentials: service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ProviderError(ProviderError { source: ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:03 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }) }))"},"target":"aws_config::meta::credentials::chain","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.394893Z","level":"INFO","fields":{"message":"external operation fetch_batch::get failed, retrying in 128ms: indeterminate: s3 get meta err: failed to construct request"},"target":"mz_persist_client::internal::machine"}
clusterd {"timestamp":"2023-06-13T20:09:04.533256Z","level":"INFO","fields":{"message":"credentials cache returned CredentialsNotLoaded, ignoring"},"target":"aws_http::auth","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.560945Z","level":"WARN","fields":{"message":"STS returned an error assuming web identity role","error":"service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:03 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }))"},"target":"aws_config::web_identity_token","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.561244Z","level":"WARN","fields":{"message":"provider failed to provide credentials","provider":"WebIdentityToken","error":"an error occurred while loading credentials: service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ProviderError(ProviderError { source: ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:03 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }) }))"},"target":"aws_config::meta::credentials::chain","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.561451Z","level":"INFO","fields":{"message":"external operation fetch_batch::get failed, retrying in 256ms: indeterminate: s3 get meta err: failed to construct request"},"target":"mz_persist_client::internal::machine"}
clusterd {"timestamp":"2023-06-13T20:09:04.813752Z","level":"INFO","fields":{"message":"credentials cache returned CredentialsNotLoaded, ignoring"},"target":"aws_http::auth","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.840466Z","level":"WARN","fields":{"message":"STS returned an error assuming web identity role","error":"service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:03 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }))"},"target":"aws_config::web_identity_token","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.840796Z","level":"WARN","fields":{"message":"provider failed to provide credentials","provider":"WebIdentityToken","error":"an error occurred while loading credentials: service error: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements: InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements (ProviderError(ProviderError { source: ServiceError(ServiceError { source: InvalidIdentityTokenException(InvalidIdentityTokenException { message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), meta: ErrorMetadata { code: Some(\"InvalidIdentityToken\"), message: Some(\"Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements\"), extras: Some({\"aws_request_id\": \" 00000000-0000-0000-0000-000000000000\"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {\"x-amzn-requestid\": \" 00000000-0000-0000-0000-000000000000\", \"content-type\": \"text/xml\", \"content-length\": \"390\", \"date\": \"Tue, 13 Jun 2023 20:09:03 GMT\", \"connection\": \"close\"}, body: SdkBody { inner: Once(Some(b\"<ErrorResponse xmlns=\\\"https://sts.amazonaws.com/doc/2011-06-15/\\\">\\n <Error>\\n <Type>Sender</Type>\\n <Code>InvalidIdentityToken</Code>\\n <Message>Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\\n </Error>\\n <RequestId> 00000000-0000-0000-0000-000000000000</RequestId>\\n</ErrorResponse>\\n\")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }) }))"},"target":"aws_config::meta::credentials::chain","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:04.841011Z","level":"INFO","fields":{"message":"external operation fetch_batch::get failed, retrying in 512ms: indeterminate: s3 get meta err: failed to construct request"},"target":"mz_persist_client::internal::machine"}
clusterd {"timestamp":"2023-06-13T20:09:05.324239Z","level":"INFO","fields":{"message":"credentials cache returned CredentialsNotLoaded, ignoring"},"target":"aws_http::auth","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T20:09:05.363141Z","level":"INFO","fields":{"message":"credentials cache miss occurred; added new AWS credentials (took 43.184745ms)"},"target":"aws_credential_types::cache::lazy_caching","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T21:09:29.655483Z","level":"INFO","fields":{"message":"credentials cache returned CredentialsNotLoaded, ignoring"},"target":"aws_http::auth","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T21:09:29.691265Z","level":"INFO","fields":{"message":"credentials cache miss occurred; added new AWS credentials (took 39.945721ms)"},"target":"aws_credential_types::cache::lazy_caching","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T22:09:39.545786Z","level":"INFO","fields":{"message":"credentials cache returned CredentialsNotLoaded, ignoring"},"target":"aws_http::auth","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T22:09:39.714404Z","level":"INFO","fields":{"message":"credentials cache miss occurred; added new AWS credentials (took 172.900715ms)"},"target":"aws_credential_types::cache::lazy_caching","span":{"name":"lazy_load_credentials"},"spans":[{"name":"lazy_load_credentials"}]}
clusterd {"timestamp":"2023-06-13T23:07:49.226265Z","level":"INFO","fields":{"message":"poisoning connection: SmithyConnection { is_proxied: false, remote_addr: Some(54.231.134.18:443) }"},"target":"aws_smithy_client::poison"}
clusterd {"timestamp":"2023-06-13T23:07:49.226334Z","level":"INFO","fields":{"message":"smithy connection was poisoned"},"target":"aws_smithy_http::connection"}
ysaito1001
rcoh
github-actions[bot]
Describe the bug
At @MaterializeInc we run a number of
clusterdprocesses that continually read and write from S3. (We make a streaming database that's in the business of reading data from S3, transforming it, and writing it back to S3.)During today's AWS outage, these
clusterdprocesses all experienced an outage. Almost all of theclusterdprocesses recovered, except for two that continually produced error messages like the following:AIUI,
rollup::setis attempting to write to S3, whiles3 get metais attempting to read from S3. The following error message (request has timed out) or (failed to construct request) comes straight from the AWS SDK, AFAICT.Expected Behavior
We expected our S3 reads/writes to eventually succeed on retry.
Current Behavior
The S3 reads/writes continued failing forever.
We validated that the two affected containers did in fact have access to S3—logging in to the containers and manually issuing S3 requests was successful at the time shown in the S3 logs.
Reproduction Steps
So sorry, but we have no reproduction of this. We run chaos tests in our CI that interrupt network connections but we've never seen anything like this. I think it's unlikely we'll see this again until another wide AWS outage.
I wonder if it's something specific about interrupting the IAM connections in the way that the AWS outage did. We don't test that chaos extensively in our CI.
Possible Solution
The reason I wanted to file this issue is because of this
printlnthat we're seeing in the output:https://github.com/awslabs/smithy-rs/blob/312d190535b1c77625d662d18313b90af64cb448/rust-runtime/aws-smithy-http/src/connection.rs#L85
This looks like a stray debugging
println. It was added in https://github.com/awslabs/smithy-rs/pull/2445. I'm just spitballing, but I'm wondering if this println is related to the issue. Perhaps connections in this process weren't getting poisoned properly because the connection metadata wasn't available?I've never seen this
printlnin our logs while debugging before. Unfortunately I can't say that we've truly never seen this log on the unaffected processes because as aprintlnrather than atracinglog it doesn't get picked up by our logging infrastructure. It's possible that thisprintlnis actually a normal occurrence when there are S3 connectivity issues, and not indicative of a failure to poison broken connections.If nothing else, seems like the debugging
printlnought to be removed!Additional Information/Context
No response
Version
Environment details (OS name and version, etc.)
Linux 5.10.178-162.673.amzn2.x86_64
Logs
No response