search

DalgoT4D / prefect-proxy

GNU Affero General Public License v3.0
0 stars 7 forks source link

allow field overrides #123

Closed fatchat closed 2 months ago

fatchat commented 2 months ago

the dbt cli profile's target-configs needs the allow_field_overrides flag to be set to True for postgres profiles

fatchat commented 2 months ago 
python scripts/allow_field_overrides.py --cli-block-name wastewarriors-dbtrepo --allow-field-overrides
python scripts/allow_field_overrides.py --cli-block-name ylabs-dbtrepo --allow-field-overrides
python scripts/allow_field_overrides.py --cli-block-name alpha1-dbtrepo --allow-field-overrides
python scripts/allow_field_overrides.py --cli-block-name delta1-dbtrepo --allow-field-overrides
python scripts/allow_field_overrides.py --cli-block-name educate-dbtrepo --allow-field-overrides
python scripts/allow_field_overrides.py --cli-block-name epsilon1-dbtrepo --allow-field-overrides
python scripts/allow_field_overrides.py --cli-block-name gamma1-dbtrepo --allow-field-overrides
python scripts/allow_field_overrides.py --cli-block-name goalkeep-dbtrepo --allow-field-overrides
python scripts/allow_field_overrides.py --cli-block-name lahi-dbtrepo --allow-field-overrides
fatchat commented 2 months ago

also created dbtrepo/profiles/ directory where missing:

fatchat commented 2 months ago

tested omega1 ... the first dbt-run failed to fetch the cli-profile block. running it again worked but why is this happening

proxy tells us

sqlalchemy.exc.InterfaceError: (sqlalchemy.dialects.postgresql.asyncpg.InterfaceError) <class 'asyncpg.exceptions._base.InterfaceError'>: connection is closed

which isn't enlightening

Ishankoradia commented 2 months ago

tested omega1 ... the first dbt-run failed to fetch the cli-profile block. running it again worked but why is this happening

proxy tells us

sqlalchemy.exc.InterfaceError: (sqlalchemy.dialects.postgresql.asyncpg.InterfaceError) <class 'asyncpg.exceptions._base.InterfaceError'>: connection is closed

which isn't enlightening

This keeps happening every now and then. With less frequency as off now. I have faced this while creating connnections (i.e. deployments).

Rishikesh63 commented 2 months ago

@fatchat i didn't understand how to tackle this problem:- File "C:\Users\User.pyenv\pyenv-win\versions\3.10.11\lib\importlib__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed
File "C:\Users\User\Desktop\DMP\prefect-proxy\proxy\main.py", line 9, in
from proxy.service import ( File "C:\Users\User\Desktop\DMP\prefect-proxy\proxy\service.py", line 36, in from proxy.prefect_flows import deployment_schedule_flow_v4 File "C:\Users\User\Desktop\DMP\prefect-proxy\proxy\prefect_flows.py", line 11, in from prefect_airbyte.flows import reset_connection ImportError: cannot import name 'reset_connection' from 'prefect_airbyte.flows' (C:\Users\User\Desktop\DMP\DDP_backend\venv\lib\site-packages\prefect_airbyte\flows.py)

Rishikesh63 commented 2 months ago

uvicorn proxy.main:app --workers 4 --host 0.0.0.0 --port 8001

after running   this code shows above error
fatchat commented 2 months ago

@Rishikesh63 you need to pip install -r requirements.txt which installs a fork of prefect_airbyte which contains the reset_connection function

Rishikesh63 commented 2 months ago

@Rishikesh63 you need to pip install -r requirements.txt which installs a fork of prefect_airbyte which contains the reset_connection function

There is already requirements.txt, prefect_airbyte and reset_connection still showing error