apache-airflow-providers-amazon - Not able to connect to minio endpoint_url - Error Could not connect to the endpoint URL: "https://sts.default.amazonaws.com/"

[mohankumar-h]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

apache-airflow-providers-amazon 8.25.0

Apache Airflow version

2.9.3

Operating System

airflow@airflow-webserver-7d758d8f79-lcpnr:/opt/airflow$ cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

Deployment

Official Apache Airflow Helm Chart

Deployment details

Deployed though official helm chart on k8s 1.29

What happened

We are trying to connect to minio s3 object store bucket using apache-airflow-providers-amazon provider Connector.

1. Login to Airflow Webserver
2. Admin -> connections -> Connection ID (Any Name you can provide )-> Connection Type as Amazon Web services -> Extra { "aws_access_key_id": "XXXXXXXXXXXXXX", "aws_secret_access_key": "XXXXXXXXXXXXXXXXXXXXX", "endpoint_url": "https://minio-url.xxx.com:443", "verify": false }

When we test connection

Error: 'EndpointConnectionError' error occurred while testing connection: Could not connect to the endpoint URL: "https://sts.default.amazonaws.com/"

It is trying to connect https://sts.default.amazonaws.com/ for authentication and authorization

Why it is trying to connect https://sts.default.amazonaws.com/ ? instead of "endpoint_url": "https://minio-url.xxx.com:443",?

What you think should happen instead

It suppose to connect to minio endpoint_url "endpoint_url": "https://minio-url.xxx.com:443", instead of connecting to https://sts.default.amazonaws.com

How to reproduce

We are trying to connect to minio s3 object store bucket using apache-airflow-providers-amazon provider Connector.

1. Login to Airflow Webserver
2. Admin -> connections -> Connection ID (Any Name you can provide )-> Connection Type as Amazon Web services -> Extra { "aws_access_key_id": "XXXXXXXXXXXXXX", "aws_secret_access_key": "XXXXXXXXXXXXXXXXXXXXX", "endpoint_url": "https://minio-url.xxx.com:443", "verify": false }

When we test connection

Error: 'EndpointConnectionError' error occurred while testing connection: Could not connect to the endpoint URL: "https://sts.default.amazonaws.com/"

It is trying to connect https://sts.default.amazonaws.com/ for authentication and authorization

Why it is trying to connect https://sts.default.amazonaws.com/ ? instead of "endpoint_url": "https://minio-url.xxx.com:443",?

Anything else

No response

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[eladkal]

duplicate of https://github.com/apache/airflow/issues/43871