search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
35.21k stars 13.76k forks source link

ODBC Provider Configuration #40405

Closed hbc-acai closed 4 days ago

hbc-acai commented 4 days ago

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.9.1

What happened?

I added environment variable AIRFLOW__PROVIDERS_ODBC__ALLOW_DRIVER_IN_EXTRA, but it seems it is not picked up by the config parser.

This is what I see:

  1. environment variable is there. It is set in the helm chart overrides. 
airflow@airflow-scheduler-69d5c977c-r85tz:/opt/airflow$ echo $AIRFLOW__PROVIDERS_ODBC__ALLOW_DRIVER_IN_EXTRA
True
  1. The list command does not return anything 
    
airflow@airflow-scheduler-69d5c977c-r85tz:/opt/airflow$ airflow config list --section providers.odbc
/home/airflow/.local/lib/python3.10/site-packages/airflow/metrics/statsd_logger.py:184 RemovedInAirflow3Warning: The basic metric validator will be deprecated in the future in favor of pattern-matching.  You can try this now by setting config option metrics_use_pattern_match to True.

3. Interestingly, the `get-value` command shows the variable

airflow@airflow-scheduler-69d5c977c-r85tz:/opt/airflow$ airflow config get-value providers.odbc allow_driver_in_extra /home/airflow/.local/lib/python3.10/site-packages/airflow/metrics/statsd_logger.py:184 RemovedInAirflow3Warning: The basic metric validator will be deprecated in the future in favor of pattern-matching. You can try this now by setting config option metrics_use_pattern_match to True. True


4. When I use `SQLSensor`, I get this warning:

[2024-06-24, 13:58:51 UTC] {odbc.py:119} WARNING - You have supplied 'driver' via connection extra but it will not be used. In order to use 'driver' from extra you must set airflow config setting allow_driver_in_extra = True in section providers.odbc. Alternatively you may specify driver via 'driver' parameter of the hook constructor or via 'hook_params' dictionary with key 'driver' if using SQL operators.


### What you think should happen instead?

_No response_

### How to reproduce

Add the following section in the helm chart overrides:

Operating System

Azure Kubernetes Service

Versions of Apache Airflow Providers

airflow@airflow-scheduler-69d5c977c-r85tz:/opt/airflow$ pip freeze |grep airflow-prov apache-airflow-providers-amazon==8.20.0 apache-airflow-providers-celery==3.6.2 apache-airflow-providers-cncf-kubernetes==8.3.0 apache-airflow-providers-common-io==1.3.1 apache-airflow-providers-common-sql==1.12.0 apache-airflow-providers-docker==3.10.0 apache-airflow-providers-elasticsearch==5.3.4 apache-airflow-providers-fab==1.0.4 apache-airflow-providers-ftp==3.8.0 apache-airflow-providers-google==10.17.0 apache-airflow-providers-grpc==3.4.1 apache-airflow-providers-hashicorp==3.6.4 apache-airflow-providers-http==4.10.1 apache-airflow-providers-imap==3.5.0 apache-airflow-providers-microsoft-azure==10.0.0 apache-airflow-providers-mysql==5.5.4 apache-airflow-providers-odbc==4.5.0 apache-airflow-providers-openlineage==1.7.0 apache-airflow-providers-postgres==5.10.2 apache-airflow-providers-redis==3.6.1 apache-airflow-providers-sendgrid==3.4.0 apache-airflow-providers-sftp==4.9.1 apache-airflow-providers-slack==8.6.2 apache-airflow-providers-smtp==1.6.1 apache-airflow-providers-snowflake==5.4.0 apache-airflow-providers-sqlite==3.7.1 apache-airflow-providers-ssh==3.10.1

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

Code of Conduct