[s3exporter] 405 returned when using the S3 exporter cross AWS accounts

Component(s)

S3 exporter

What happened?

Description

We're noticing errors in the opentelemetry-collector pod logs when trying to use the S3 exporter cross AWS accounts. e.g. when trying to collect logs from pods running in an EKS cluster in a given AWS account and send those to an S3 bucket located in a different AWS account. The error:

"error": "WebIdentityErr: failed to retrieve credentials\ncaused by: SerializationError: failed to unmarshal error message\n\tstatus code: 405, request id: \ncaused by: UnmarshalError: failed to unmarshal error message
…
caused by: unknown error response tag.

In AWS, when with S3 access logging, we can see the below error:

...REST.POST.OBJECT .../ "POST /.../ HTTP/1.1" 405 MethodNotAllowed....

Yet I've double checked my AWS IAM role, policy and trust relationship, and they look correct (actually using a similar role and policy does work when we're in the same AWS account).

Steps to Reproduce

Use the below config:

config:
  exporters:
    awss3:
      s3uploader:
        endpoint: "https://<bucket-name>.s3.<region>.amazonaws.com/<folder>"
        region: "<region>"
        s3_prefix: "logs"
        s3_partition: "hour"
        role_arn: "arn:aws:iam::<aws-account-id>:role/<aws-iam-role-name>"

Expected Result

No 405 response, and logs should appear in the S3 bucket.

Actual Result

405 response, no logs in the bucket

Additional information

Considering the AWS Go SDK v1 deprecation next year, it might be worth switching to v2 if possible. Related discussion.

Collector version

opentelemetry-collector-contrib:0.97.0

Environment information

Environment

EKS

OpenTelemetry Collector configuration

config:
  exporters:
    awss3:
      s3uploader:
        endpoint: "https://<bucket-name>.s3.<region>.amazonaws.com/<folder>"
        region: "<region>"
        s3_prefix: "logs"
        s3_partition: "hour"
        role_arn: "arn:aws:iam::<aws-account-id>:role/<aws-iam-role-name>"

Log output

See above.

Additional context

None

open-telemetry / opentelemetry-collector-contrib