"boto3 >= 1.26.157 required" error in redshift-connector 2.0.912

[guru-pochineni]

Driver version

2.0.912

Redshift version

PostgreSQL 8.0.2 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.4.2 20041017 (Red Hat 3.4.2-6.fc3), Redshift 1.0.53064

Client Operating System

Amazon Linux

Python version

3.8

Table schema

NA

Problem description

We use apache-airflow-providers-amazon==3.4.0 to connect to Redshift via Airflow using IAM auth. This airflow package installs the latest version of redshift-connector by default. We are facing issues connecting to Redshift in 2.0.912.

Error message: ModuleNotFoundError: boto3 >= 1.26.157 required for authentication with Amazon Redshift using custom domain name. Please upgrade the installed version of boto3 to use this functionality.

Due to some legacy libraries our boto3 version is less 1.26.157, which is causing connection to fail. This was working as expected till 2.0.911. I see a recent change for "Support Redshift custom domain name" has added this check. As we are using IAM auth, we supply cluster_identifier in Airflow which I think is now considered as cname?

Is this an expected change?

Airflow connection details

- id: redshift_connection
      type: redshift
      schema: testdb
      port: 5439
      extra: '{"iam": true, "region": "us-east-1", "cluster_identifier": "redshift-warehouse", "db_user": "redshift_test_user"}'

Python Driver trace logs

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/decorators/base.py", line 134, in execute
    return_value = super().execute(context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 150, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 161, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/opt/airflow/dag.py", line 119, in run_dag
    redshift_hook.run(sql)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/hooks/dbapi.py", line 177, in run
    with closing(self.get_conn()) as conn:
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/hooks/redshift_sql.py", line 133, in get_conn
    conn: RedshiftConnection = redshift_connector.connect(**conn_kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/redshift_connector/__init__.py", line 336, in connect
    IamHelper.set_iam_properties(info)
  File "/home/airflow/.local/lib/python3.8/site-packages/redshift_connector/iam_helper.py", line 117, in set_iam_properties
    raise ModuleNotFoundError(
ModuleNotFoundError: boto3 >= 1.26.157 required for authentication with Amazon Redshift using custom domain name. Please upgrade the installed version of boto3 to use this functionality.

Reproduction code

Connect via Airflow with boto3 < 1.26.157.

[Brooke-white]

Hi @guru-pochineni, thank you for bringing this issue to our attention. The root cause of this issue has been identified, it is in iam_helper.py LOC 116. We have identified a fix and will include in our next release which is scheduled to occur in August.

In the meantime, my recommendation would be to downgrade the version of redshift-connector in use. We apologies for the inconvenience. If downgrading redshift-connector or waiting until August for the fix is not acceptable please let me know and we can make an ad-hoc release to address this bug. Please install the latest version of redshift-connector, it should be fixed now. Thank you for your understanding!

[guru-pochineni]

Hello @Brooke-white, thank you for promptly fixing this and doing an ad hoc release. I can connect to Redshift using the latest version of redshift-connector.

Closing this issue.