search

astronomer / airflow-dbt-demo

A repository of sample code to accompany our blog post on Airflow and dbt.
166 stars 65 forks source link

dags don't run in demo #8

Open dschneiderch opened 2 years ago

dschneiderch commented 2 years ago

I'm a trying to better understand how airflow and dbt can be used together but I can't actually run the demo. when I trigger dbt_basic_dag I get airflow.exceptions.AirflowNotFoundException: The conn_id `***` isn't defined so i appears that the docker container doesn't have the environment variables defined needed to connect to postgres. is there a simple fix for this?

dschneiderch commented 2 years ago

The output from astro dev start suggests the u/p should be admin/admin but that doesn't seem to work:

Airflow Webserver: http://localhost:8080
Postgres Database: localhost:5432/postgres
The default credentials are admin:admin
~/b/i/airflow-dbt-demo ❯❯❯ pgcli -h localhost -d postgres -u admin                                                                                           
Password for admin:
connection to server at "localhost" (127.0.0.1), port 5432 failed: FATAL:  password authentication failed for user "admin"
bmonteiro commented 1 year ago

The output from astro dev start suggests the u/p should be admin/admin but that doesn't seem to work:

Airflow Webserver: http://localhost:8080
Postgres Database: localhost:5432/postgres
The default credentials are admin:admin
~/b/i/airflow-dbt-demo ❯❯❯ pgcli -h localhost -d postgres -u admin                                                                                           
Password for admin:
connection to server at "localhost" (127.0.0.1), port 5432 failed: FATAL:  password authentication failed for user "admin"

Project is running! All components are now available.

Airflow Webserver: http://localhost:8080 Postgres Database: localhost:5432/postgres The default Airflow UI credentials are: admin:admin The default Postgres DB credentials are: postgres:postgres

===> postgres:postgres

pgcli -h localhost -d postgres -u postgres Password for postgres: Server: PostgreSQL 12.6 (Debian 12.6-1.pgdg100+1) Version: 3.5.0 Home: http://pgcli.com postgres@localhost:postgres>

JKCai commented 1 year ago

@dschneiderch Not sure if you've fixed this, by manually adding connection, it works for me. However, a new error pops up and not sure if others know how to fix. 

[2022-12-01, 16:23:03 AEDT] {subprocess.py:75} INFO - Running command: ['/bin/bash', '-c', 'dbt seed --profiles-dir /usr/local/airflow/dbt --project-dir /usr/local/airflow/dbt']
[2022-12-01, 16:23:03 AEDT] {subprocess.py:86} INFO - Output:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 05:23:05  Error importing adapter: No module named 'dbt.adapters.*****'
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 05:23:05  Encountered an error while reading profiles:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 05:23:05    ERROR: Runtime Error
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   Credentials in profile "jaffle_shop", target "dev" invalid: Runtime Error
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -     Could not find adapter type *****!
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 05:23:05  Defined profiles:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 05:23:05   - jaffle_shop
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 05:23:05
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - For more information on configuring profiles, please consult the dbt docs:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - https://docs.getdbt.com/docs/configure-your-profile
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 05:23:05  Encountered an error:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - Runtime Error
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   Could not run dbt
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 05:23:05  Traceback (most recent call last):
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/adapters/factory.py", line 62, in load_plugin
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -     mod: Any = import_module("." + name, "dbt.adapters")
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -     return _bootstrap._gcd_import(name[level:], package, level)
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - ModuleNotFoundError: No module named 'dbt.adapters.*****'
JKCai commented 1 year ago

For anyone who failed to run due to connection error, you should set hostname to postgres instead of localhost in airflow connection. I now can run this demo.

chenna09 commented 1 year ago

@dschneiderch Not sure if you've fixed this, by manually adding connection, it works for me. However, a new error pops up and not sure if others know how to fix.

[2022-12-01, 16:23:03 AEDT] {subprocess.py:75} INFO - Running command: ['/bin/bash', '-c', 'dbt seed --profiles-dir /usr/local/airflow/dbt --project-dir /usr/local/airflow/dbt']
[2022-12-01, 16:23:03 AEDT] {subprocess.py:86} INFO - Output:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Error importing adapter: No module named 'dbt.adapters.*****'
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Encountered an error while reading profiles:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05    ERROR: Runtime Error
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   Credentials in profile "jaffle_shop", target "dev" invalid: Runtime Error
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -     Could not find adapter type *****!
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Defined profiles:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05   - jaffle_shop
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - For more information on configuring profiles, please consult the dbt docs:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - https://docs.getdbt.com/docs/configure-your-profile
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - 
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Encountered an error:
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - Runtime Error
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   Could not run dbt
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - �[0m05:23:05  Traceback (most recent call last):
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/site-packages/dbt/adapters/factory.py", line 62, in load_plugin
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -     mod: Any = import_module("." + name, "dbt.adapters")
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -     return _bootstrap._gcd_import(name[level:], package, level)
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO -   File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
[2022-12-01, 16:23:05 AEDT] {subprocess.py:93} INFO - ModuleNotFoundError: No module named 'dbt.adapters.*****'

add dbt-postgres adapter in the requirement file, it fixed my issue. dbt-postgres==1.3.0