[Bug] dbtCore - after upgrading, can't run DBT

moseleyi commented 2 months ago

Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I installed dbt-core, bigquery, and postgres, and then when typing dbt --version I get this error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\___\AppData\Roaming\Python\Python312\Scripts\dbt.exe\__main__.py", line 4, in <module>
  File "C:\Users\___t\AppData\Roaming\Python\Python312\site-packages\dbt\cli\__init__.py", line 1, in <module>
    from .main import cli as dbt_cli  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\___\AppData\Roaming\Python\Python312\site-packages\dbt\cli\main.py", line 14, in <module>
    from dbt.cli import requires, params as p
  File "C:\Users\___\AppData\Roaming\Python\Python312\site-packages\dbt\cli\requires.py", line 10, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
ModuleNotFoundError: No module named 'dbt.adapters.factory'

This is after installing bigquery, postgres,

Expected Behavior

Should show the version of dbt packages

Steps To Reproduce

Install dbt-core
Run dbt --version

Relevant log output

No response

Environment

- OS: Windows
- Python: 3.12
- dbt: 1.8

Which database adapter are you using with dbt?

No response

Additional Context

No response

jtcohen6 commented 2 months ago

@moseleyi Could you run pip freeze | grep 'dbt' and report back with the exact versions of Python packages that you have installed?

Also - you might try installing all of these by name, along with the --upgrade flag, and see if that resolves the issue for you:

pip install --upgrade dbt-core dbt-bigquery dbt-postgres

rubenhelsloot commented 2 months ago

@jtcohen6 I have the same issue, this is the output you asked for:

❯ pip freeze | grep 'dbt'
dbt-adapters==1.1.1
dbt-common==1.0.4
dbt-core==1.8.0
dbt-extractor==0.5.1
dbt-postgres==1.8.0
dbt-semantic-interfaces==0.5.1
sqlfluff-templater-dbt==3.0.6

The issue persists when I uninstall and then reinstall dbt-core and dbt-postgres. FWIW I'm using poetry.

jtcohen6 commented 2 months ago

Thanks @rubenhelsloot! Are you also using Python 3.12?

moseleyi commented 2 months ago

My output:

dbt-adapters==1.1.1
dbt-common==1.0.4
dbt-core==1.8.0
dbt-extractor==0.5.1
dbt-postgres==1.8.0
dbt-semantic-interfaces==0.5.1

Some people in my team got it working but maybe because it was clean install, they never had to use python packages before. Perhaps some dependencies of dbt are not updated with the latest update?

moseleyi commented 2 months ago

Surprisingly dbt 1.7.13 works fine with Python 3.12.3 - I checked on my other laptop

dbeatty10 commented 2 months ago

@rubenhelsloot! Are you also using Windows?

@moseleyi Which commands are you using to installing dbt? Are you using a virtual environment with venv or Poetry?

The following files and commands worked for me:

on macOS
using venv
with Python 3.12.3 (managed by pyenv)

Create this file:

requirements.txt

dbt-adapters==1.1.1
dbt-common==1.0.4
dbt-core==1.8.0
dbt-extractor==0.5.1
dbt-postgres==1.8.0
dbt-semantic-interfaces==0.5.1

Run these commands:

python3 -m venv env
source env/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
source env/bin/activate
python --version
dbt --version
deactivate

Output:

Python 3.12.3
Core:
  - installed: 1.8.0
  - latest:    1.8.0 - Up to date!

Plugins:
  - postgres: 1.8.0 - Up to date!

leoebfolsom commented 2 months ago

In case helpful, when I upgraded to dbt-core 1.8, I had the same issue on Python 3.9.6 (OS: Mac Ventura 13.4.1), and it was resolved on upgrade to Python 3.12.3.

moseleyi commented 2 months ago

I don't use venv nor poetry. At first I just upgraded pip install dbt-core --upgrade and the adapters. That didn't work so I tried uninstalling everything and installing again using different commands (core + adapters in one command, adapter only etc.)

All of them produce this error

troyharvey commented 2 months ago

Reproduced with Python 3.10.

＄ python --version
Python 3.10.14

＄ dbt deps
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ds-redshift/bin/dbt", line 5, in <module>
    from dbt.cli.main import cli
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ds-redshift/lib/python3.10/site-packages/dbt/cli/__init__.py", line 1, in <module>
    from .main import cli as dbt_cli  # noqa
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ds-redshift/lib/python3.10/site-packages/dbt/cli/main.py", line 14, in <module>
    from dbt.cli import requires, params as p
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ds-redshift/lib/python3.10/site-packages/dbt/cli/requires.py", line 10, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
ModuleNotFoundError: No module named 'dbt.adapters.factory'

paulbakkerbloom commented 2 months ago

Had the same issue with Python 3.9.18 -> solved when I upgraded to Python 3.11.6

I use poetry and upgraded through poetry update

dbt-adapters                          1.1.1             The set of adapter ...
dbt-bigquery                          1.8.0             The Bigquery adapte...
dbt-common                            1.0.4             The shared common u...
dbt-core                              1.8.0             With dbt, data anal...
dbt-extractor                         0.5.1             A tool to analyze a...
dbt-semantic-interfaces               0.5.1             The shared semantic...
dbt-tests-adapter                     1.8.0             The set of reusable...

moseleyi commented 2 months ago

I was hoping it was about python issue but seems like it's not. It could be a dependency that should be updated with dbt core update but isn't being forced to update?

moseleyi commented 2 months ago

I also found this: https://github.com/AltimateAI/vscode-dbt-power-user/issues/1029 but we all have dbt-adapters installed

brabster commented 2 months ago

A bit more info: I had the same problem, I can reproduce at will, Python 3.11.8:

Install dbt-bigquery==1.7.7 into a clean venv, working fine (via pip install -U -r requirements.txt)

Bump to dbt-bigquery>=1.7.7 and same pip command, broken

 File "/workspaces/pypi_vulnerabilities/venv/lib/python3.11/site-packages/dbt/cli/requires.py", line 10, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
ModuleNotFoundError: No module named 'dbt.adapters.factory'

Delete venv, same requirements.txt, pip install - get 1.8, works fine.

In the broken venv, the factory.py (and other) files in dbt/adapters is missing.

You should be able to reproduce these steps exactly with a codespace off https://github.com/brabster/pypi_vulnerabilities

dbeatty10 commented 2 months ago

Thanks for this info @brabster 🏅

Could you share the contents of your requirements.txt file also?

brabster commented 2 months ago

@dbeatty10 It's just the dbt-bigquery dependency. Originally as per repo. https://github.com/brabster/pypi_vulnerabilities/blob/main/requirements.txt

Clean venv, set to dbt-bigquery==1.7.7 to force pre-1.8.0 and install. Works. Then set to dbt-bigquery>=1.7.7 to upgrade to latest. Broken Clean venv, install same requirements. Works

jtcohen6 commented 2 months ago

@brabster If in step (2), you run pip install --upgrade dbt-core dbt-common dbt-adapters dbt-bigquery, do you still see the error?

(I'm hoping this might be because the underlying dependencies are not being upgraded, when they should be, but I'm not 100% positive: https://github.com/dbt-labs/dbt-core/pull/10123)

dbeatty10 commented 2 months ago

Thank you @brabster ! 🏅 I was able to reproduce the error using your description. See "reprex" below for full details.

@jtcohen6 when I tried running pip install --upgrade dbt-core dbt-common dbt-adapters dbt-bigquery, it still gave the same error.

### Reprex

Create requirements files for both 1.7 and 1.8: ```shell cat << EOF > requirements_1.7.txt dbt-bigquery==1.7.7 EOF cat << EOF > requirements_1.8.txt dbt-adapters==1.1.1 dbt-common==1.0.4 dbt-core==1.8.0 dbt-extractor==0.5.1 dbt-bigquery==1.8.0 dbt-semantic-interfaces==0.5.1 EOF ``` ✅ Create a fresh virtual environment and install the requirements for 1.7: ```shell rm -rf env python3 -m venv env source env/bin/activate python -m pip install -r requirements_1.7.txt source env/bin/activate dbt --version ``` ❌ Try upgrading by installing the requirements for 1.8 in the same virtual environment: ```shell python -m pip install -r requirements_1.8.txt source env/bin/activate dbt --version ``` Get an error like the following: ``` from dbt.adapters.factory import adapter_management, register_adapter, get_adapter ModuleNotFoundError: No module named 'dbt.adapters.factory' ``` ✅ Try upgrading by installing the requirements for 1.8 in a fresh virtual environment: ```shell deactivate rm -rf env python3 -m venv env source env/bin/activate python -m pip install -r requirements_1.8.txt source env/bin/activate dbt --version ```

dbeatty10 commented 2 months ago

When I got the same error as the reported above, the following workaround resolved it for me:

python -m pip uninstall -y dbt-adapters
python -m pip install --upgrade dbt-core dbt-common dbt-adapters
dbt --version

The pip uninstall -y dbt-adapters is the important part.

The 2nd command can be replaced with whichever command you use to install the requirements for dbt 1.8.

moseleyi commented 2 months ago

Yes, this workaround seems to work. But what is the underlying issue?

brabster commented 2 months ago

Seems to be related to the move of dbt-adapters from part of dbt-core to standalone. I can upgrade no problem from a 1.8 pre-release to latest, but anything from before dbt-adapters being standalone and boom.

I can see in verbose pip install logs that it's going around deleting the files in dbt/adapters/... but no evidence that it knows to replace them. Weird. I wonder if it's something to do with going to a standalone dependency but having all the files end up in the same place, maybe a kind of race condition so they are already there when it's checking what to install, but then it goes and deletes them afterwards leaving you broken. Perhaps moving the standalone adapters directory tree away from where it used to be could resolve the problem? Not had chance to check

dbeatty10 commented 2 months ago

@moseleyi I don't know the underlying root cause.

But I'm able to see some differences when I create two different virtual environments:

env1 is the environment that has the error
env2 is the environment that is working as expected

Difference 1

One of the differences is in this file: lib/python3.12/site-packages/dbt/__init__.py

env1 has contents that are identical to dbt-postgres dbt/__init__.py
env2 has contents that are identical to dbt-adapters dbt/__init__.py
Neither environment has contents identical to dbt-core core/dbt/__init__.py

Difference 2

env2 has the following files/folders within lib/python3.12/site-packages/dbt/adapters that are not in env1:

__init__.py
cache.py
capability.py
factory.py
protocol.py
reference_keys.py
sql

These are the files/folders that are within both:

- `__about__.py` - `__pycache__` - `base` - `clients` - `contracts` - `events` - `exceptions` - `postgres` - `py.typed` - `relation_configs` - `utils.py`

dbeatty10 commented 2 months ago

Here's another command that resolved the error for me:

python -m pip install --force-reinstall dbt-adapters

brabster commented 2 months ago

For info pip install -U --force-reinstall -r requirements.txt does not work, unfortunately.

brabster commented 2 months ago

Following previous comments - I think it might be a bug in pip? Seems like the last thing pip does is uninstall the old versions including the old dbt-core, which used to contain the adapters directory and doesn't anymore. If it's deleting the common files with the new dbt-adapters package when it does that uninstall, that might explain the missing files when you try to upgrade from core to standalone adapters

Installing collected packages: daff, dbt-semantic-interfaces, dbt-common, dbt-adapters, dbt-core, dbt-bigquery
  Attempting uninstall: dbt-semantic-interfaces
    Found existing installation: dbt-semantic-interfaces 0.4.4
    Uninstalling dbt-semantic-interfaces-0.4.4:
      Successfully uninstalled dbt-semantic-interfaces-0.4.4
  Attempting uninstall: dbt-core
    Found existing installation: dbt-core 1.7.14
    Uninstalling dbt-core-1.7.14:
      Successfully uninstalled dbt-core-1.7.14
  Attempting uninstall: dbt-bigquery
    Found existing installation: dbt-bigquery 1.7.7
    Uninstalling dbt-bigquery-1.7.7:
      Successfully uninstalled dbt-bigquery-1.7.7
Successfully installed daff-1.3.46 dbt-adapters-1.1.1 dbt-bigquery-1.8.1 dbt-common-1.0.4 dbt-core-1.8.0 dbt-semantic-interfaces-0.5.1

jkleinkauff commented 2 months ago

Same problem here. @dbeatty10 's solution worked for me.

python -m pip uninstall -y dbt-adapters
python -m pip install --upgrade dbt-core dbt-common dbt-adapters
dbt --version

brabster commented 2 months ago

I had a look around in the pip source code and if I'm understanding this correctly, it's clear what the problem is and any upgrade from pre-standalone adapters to standalone adapters will break in this way.

After interrogating PyPI for the latest versions of things, we get this (verbose) log line like this:

Installing collected packages: daff, dbt-semantic-interfaces, dbt-common, dbt-adapters, dbt-core, dbt-bigquery

That's from line 54 here which enumerates to_install - an OrderedDict, so the packages are always in alpha order, and dbt-adapters comes before dbt-core (demonstrated in the debug line I showed).

Now on line 61, pip loops over the ordered requirements, i.e. dbt-adapters first and then dbt-core, and for each package uninstalls the previous version and installs the new version.

Because dbt-core follows dbt-adapters, the files they have in common get deleted as part of the dbt-core uninstall and leaves a broken installation.

It's certainly possible I'm wrong, but I attach a verbose upgrade log and you can see the ordering and uninstall activity happening in a manner consistent with this explanation towards the end.

pip.log

It also explains every reported behaviour in this issue:

any attempt to update dbt-core in place with an existing version of dbt-core that included the adapter file structure will cause this break
cleaning the venv first will avoid the reinstall behaviour and be successful
--force-reinstall flag will make pip copy dbt-adapters back into place, fixing the break

If I can find an old version of dbt-core that didn't have those adapters files in that specific location, then I think an upgrade should work, if my theory is correct. Will let you know if I can verify that.

===== Update

Seems that adapters structure has been there since the first version of dbt-core in PyPI. Instead, I installed dbt-core==1.7.0 into a fresh venv, then hacked the site-packages/dbt-core...dist-info/RECORD file to remove all mention of anything in dbt/adapters. Then I pip install --upgrade to latest and the adapter files are there. It still breaks, but you can see the factory module is no longer causing the problem, all seems to suggest the theory is correct.

Installing collected packages: daff, sqlparse, protobuf, dbt-semantic-interfaces, dbt-common, dbt-adapters, dbt-core
  Attempting uninstall: sqlparse
    Found existing installation: sqlparse 0.4.4
    Uninstalling sqlparse-0.4.4:
      Successfully uninstalled sqlparse-0.4.4
  Attempting uninstall: protobuf
    Found existing installation: protobuf 5.26.1
    Uninstalling protobuf-5.26.1:
      Successfully uninstalled protobuf-5.26.1
  Attempting uninstall: dbt-semantic-interfaces
    Found existing installation: dbt-semantic-interfaces 0.4.4
    Uninstalling dbt-semantic-interfaces-0.4.4:
      Successfully uninstalled dbt-semantic-interfaces-0.4.4
  Attempting uninstall: dbt-core
    Found existing installation: dbt-core 1.7.0
    Uninstalling dbt-core-1.7.0:
      Successfully uninstalled dbt-core-1.7.0
Successfully installed daff-1.3.46 dbt-adapters-1.1.1 dbt-common-1.0.4 dbt-core-1.8.0 dbt-semantic-interfaces-0.5.1 protobuf-4.25.3 sqlparse-0.5.0
(venv) (venv) @brabster ➜ /workspaces/pypi_vulnerabilities (main) $ dbt debug
Traceback (most recent call last):
  File "/workspaces/pypi_vulnerabilities/venv/bin/dbt", line 5, in <module>
    from dbt.cli.main import cli
  File "/workspaces/pypi_vulnerabilities/venv/lib/python3.11/site-packages/dbt/cli/__init__.py", line 1, in <module>
    from .main import cli as dbt_cli  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/pypi_vulnerabilities/venv/lib/python3.11/site-packages/dbt/cli/main.py", line 14, in <module>
    from dbt.cli import requires, params as p
  File "/workspaces/pypi_vulnerabilities/venv/lib/python3.11/site-packages/dbt/cli/requires.py", line 10, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
  File "/workspaces/pypi_vulnerabilities/venv/lib/python3.11/site-packages/dbt/adapters/factory.py", line 21, in <module>
    from dbt.include.global_project import (
ImportError: cannot import name 'PACKAGE_PATH' from 'dbt.include.global_project' (unknown location)

Not sure how come dbt-adapters ends up in a common directory structure as dbt-core but strikes me as not ideal as it means it's not really independent. Any oversight and chance collision could cause weird problems on update. Setting dbt-adapters to install where I'd have expected it in its own director under site packages should solve the problem and avoid any regression?

naveenwashere commented 2 months ago

pip install --force-reinstall dbt-adapters

This worked for me. Thanks @dbeatty10 .

karenderer commented 2 months ago

I found a way to upgrade to 1.8 with poetry by downgrading dbt-adapters to 1.1.0 and then updating it to 1.1.1 but it's not ideal.

Details

$ poetry add --group dev dbt-core=^1.8.1 dbt-snowflake=^1.8.2 $ dbt --version > `ModuleNotFoundError: No module named 'dbt.adapters.factory'` $ poetry add --group dev dbt-adapters = "^1.1.0" change the following in pyproject.toml * set dbt-adapters to 1.1.0 * set dbt-core to 1.8.0 * set dbt-snowflake to 1.8.0 $ poetry lock --no-update $ poetry install --sync --no-root $ dbt version > works $ dbt debug > `TypeError: __post_serialize__() got an unexpected keyword argument 'context'` change the following in pyproject.toml * set dbt-adapters to 1.1.1 $ poetry lock --no-update $ poetry install --sync --no-root $ dbt version > works $ dbt debug > runs successfully

MikeMccree commented 2 months ago

pip install --force-reinstall dbt-adapters
This worked for me. Thanks @dbeatty10 .

+1 worked for me as well.

I am on mac, ran the following: pip install --upgrade \ dbt-core \ dbt-postgres \ dbt-redshift \ dbt-spark\ dbt-athena-community\ dbt-glue

Encountered the error with adapters. ran the force-reinstall and we all good.. Thanks @dbeatty10

rizahmeds commented 1 month ago

Upgraded python version from 3.11.5 to 3.12.3, and it worked.

dun-x commented 1 month ago

Here's another command that resolved the error for me:
python -m pip install --force-reinstall dbt-adapters

This work for me

elyobo commented 1 month ago

For those using poetry, my fix was to remove the env and reinstall, e.g.

poetry env remove 3.12
poetry install

nicolefeatherby commented 1 month ago

Ran into this too using a venv on python 3.12.0

pip install --force-reinstall dbt-adapters worked for me too

Got this message while doing so, but everything works, so not sure if relevant

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
snowflake-connector-python 3.5.0 requires urllib3<2.0.0,>=1.21.1, but you have urllib3 2.2.1 which is incompatible.

Evelyn-yx-Li commented 1 month ago

Hi team, is there any plan to resolve the issue without entering pip install --force-reinstall dbt-adapters? Because we're currently using pip install within the GitHub Action. Just wondering if we need to add the commands within the workflow files. Thank you!

dbeatty10 commented 1 month ago

@Evelyn-yx-Li We're not aware of anything we can do about this that wouldn't risk breaking other things. So if you're getting this error in your GitHub Actions, then we'd recommend adding a workaround to handle it.

elyobo commented 1 month ago

@Evelyn-yx-Li it seems that this is explicitly an upgrading problem, so possibly it's related to caching dependency installs e.g. if you're doing so via actions/setup-python; maybe try disabling if so?

That said, we're caching poetry dependencies in that way and haven't run into issues, it was just local developer installs that seemed to have problems.

brabster commented 1 month ago

@elyobo confident this is the underlying reason https://github.com/dbt-labs/dbt-core/issues/10135#issuecomment-2118230255. If so it is explicitly an upgrading problem, because of the order in which packages are cleaned up and dbt's placing multiple modules into a shared directory structure. Explains the variety of issues reported.

Evelyn-yx-Li commented 1 month ago

Hi, I've tried the solutions above in our prod, but it shows "command not found". Both force-reinstall and uninstall dbt-adapters. But it works well within local and CI.

------------------------------------------------------------------------------------------------------------------------
DBT VERSION
------------------------------------------------------------------------------------------------------------------------
Checking DBT version...
dbt -v
./dbt-forklift.sh: line 49: dbt: command not found

------------------------------------------------------------------------------------------------------------------------
SET UP DBT
------------------------------------------------------------------------------------------------------------------------
Installing DBT dependencies...
pip uninstall -y dbt-adapters
WARNING: Skipping dbt-adapters as it is not installed.
pyenv: cannot rehash: /usr/local/.pyenv/shims isn't writable
pip install dbt-core dbt-bigquery==1.8.1 dbt-redshift==1.8.0 dbt-snowflake==1.8.3
Defaulting to user installation because normal site-packages is not writeable
Collecting dbt-core
  Using cached dbt_core-1.8.2-py3-none-any.whl.metadata (3.9 kB)
Collecting dbt-bigquery==1.8.1
  Using cached dbt_bigquery-1.8.1-py3-none-any.whl.metadata (3.2 kB)
Collecting dbt-redshift==1.8.0
  Using cached dbt_redshift-1.8.0-py3-none-any.whl.metadata (3.2 kB)
Collecting dbt-snowflake==1.8.3
  Using cached dbt_snowflake-1.8.3-py3-none-any.whl.metadata (3.2 kB)
Collecting dbt-common<2.0,>=1.0.4 (from dbt-bigquery==1.8.1)
  Using cached dbt_common-1.3.0-py3-none-any.whl.metadata (5.3 kB)
Collecting dbt-adapters<2.0,>=1.1.1 (from dbt-bigquery==1.8.1)
  Using cached dbt_adapters-1.2.1-py3-none-any.whl.metadata (2.5 kB)
Collecting google-cloud-bigquery<4.0,>=3.0 (from google-cloud-bigquery[pandas]<4.0,>=3.0->dbt-bigquery==1.8.1)
  Using cached google_cloud_bigquery-3.24.0-py2.py3-none-any.whl.metadata (8.9 kB)
Collecting google-cloud-storage~=2.4 (from dbt-bigquery==1.8.1)
  Using cached google_cloud_storage-2.17.0-py2.py3-none-any.whl.metadata (6.6 kB)
Collecting google-cloud-dataproc~=5.0 (from dbt-bigquery==1.8.1)
  Using cached google_cloud_dataproc-5.9.3-py2.py3-none-any.whl.metadata (5.2 kB)
Collecting google-api-core>=2.11.0 (from dbt-bigquery==1.8.1)
  Using cached google_api_core-2.19.0-py3-none-any.whl.metadata (2.7 kB)
Collecting dbt-postgres~=1.8.0 (from dbt-redshift==1.8.0)
  Using cached dbt_postgres-1.8.1-py3-none-any.whl.metadata (3.4 kB)
Collecting redshift-connector!=2.0.914,<2.0.918,>=2.0.913 (from dbt-redshift==1.8.0)
  Using cached redshift_connector-2.0.917-py3-none-any.whl.metadata (66 kB)
Collecting sqlparse<0.6.0,>=0.5.0 (from dbt-redshift==1.8.0)
  Using cached sqlparse-0.5.0-py3-none-any.whl.metadata (3.9 kB)
Collecting agate (from dbt-redshift==1.8.0)
  Using cached agate-1.11.0-py2.py3-none-any.whl.metadata (3.2 kB)
Collecting snowflake-connector-python~=3.0 (from snowflake-connector-python[secure-local-storage]~=3.0->dbt-snowflake==1.8.3)
  Using cached snowflake_connector_python-3.10.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (62 kB)
Collecting agate (from dbt-redshift==1.8.0)
  Using cached agate-1.9.1-py2.py3-none-any.whl.metadata (3.2 kB)
Collecting Jinja2<4,>=3.1.3 (from dbt-core)
  Using cached jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
Collecting mashumaro<4.0,>=3.9 (from mashumaro[msgpack]<4.0,>=3.9->dbt-core)
  Using cached mashumaro-3.13-py3-none-any.whl.metadata (114 kB)
Collecting logbook<1.6,>=1.5 (from dbt-core)
  Using cached Logbook-1.5.3.tar.gz (85 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting click<9.0,>=8.0.2 (from dbt-core)
  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting networkx<4.0,>=2.3 (from dbt-core)
  Using cached networkx-3.3-py3-none-any.whl.metadata (5.1 kB)
Collecting protobuf<5,>=4.0.0 (from dbt-core)
  Using cached protobuf-4.25.3-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes)
Collecting requests<3.0.0 (from dbt-core)
  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting pathspec<0.13,>=0.9 (from dbt-core)
  Using cached pathspec-0.12.1-py3-none-any.whl.metadata (21 kB)
Collecting dbt-extractor<=0.6,>=0.5.0 (from dbt-core)
  Using cached dbt_extractor-0.5.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
Collecting minimal-snowplow-tracker<0.1,>=0.0.2 (from dbt-core)
  Using cached minimal-snowplow-tracker-0.0.2.tar.gz (12 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting dbt-semantic-interfaces<0.6,>=0.5.1 (from dbt-core)
  Using cached dbt_semantic_interfaces-0.5.1-py3-none-any.whl.metadata (2.6 kB)
Collecting packaging>20.9 (from dbt-core)
  Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
Collecting pytz>=2015.7 (from dbt-core)
  Using cached pytz-2024.1-py2.py3-none-any.whl.metadata (22 kB)
Collecting pyyaml>=6.0 (from dbt-core)
  Using cached PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting daff>=1.3.46 (from dbt-core)
  Using cached daff-1.3.46.tar.gz (149 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting typing-extensions>=4.4 (from dbt-core)
  Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Collecting Babel>=2.0 (from agate->dbt-redshift==1.8.0)
  Using cached Babel-2.15.0-py3-none-any.whl.metadata (1.5 kB)
Collecting isodate>=0.5.4 (from agate->dbt-redshift==1.8.0)
  Using cached isodate-0.6.1-py2.py3-none-any.whl.metadata (9.6 kB)
Collecting leather>=0.3.2 (from agate->dbt-redshift==1.8.0)
  Using cached leather-0.4.0-py2.py3-none-any.whl.metadata (2.8 kB)
Collecting parsedatetime!=2.5,>=2.1 (from agate->dbt-redshift==1.8.0)
  Using cached parsedatetime-2.6-py3-none-any.whl.metadata (4.7 kB)
Collecting python-slugify>=1.2.1 (from agate->dbt-redshift==1.8.0)
  Using cached python_slugify-8.0.4-py2.py3-none-any.whl.metadata (8.5 kB)
Collecting pytimeparse>=1.1.5 (from agate->dbt-redshift==1.8.0)
  Using cached pytimeparse-1.1.8-py2.py3-none-any.whl.metadata (3.4 kB)
Collecting colorama<0.5,>=0.3.9 (from dbt-common<2.0,>=1.0.4->dbt-bigquery==1.8.1)
  Using cached colorama-0.4.6-py2.py3-none-any.whl.metadata (17 kB)
Collecting jsonschema<5.0,>=4.0 (from dbt-common<2.0,>=1.0.4->dbt-bigquery==1.8.1)
  Using cached jsonschema-4.22.0-py3-none-any.whl.metadata (8.2 kB)
Collecting python-dateutil<3.0,>=2.0 (from dbt-common<2.0,>=1.0.4->dbt-bigquery==1.8.1)
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting psycopg2<3.0,>=2.9 (from dbt-postgres~=1.8.0->dbt-redshift==1.8.0)
  Using cached psycopg2-2.9.9.tar.gz (384 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error

  Ã Getting requirements to build wheel did not run successfully.
  â exit code: 1
  â°â> [21 lines of output]
      running egg_info
      writing psycopg2.egg-info/PKG-INFO
      writing dependency_links to psycopg2.egg-info/dependency_links.txt
      writing top-level names to psycopg2.egg-info/top_level.txt

      Error: pg_config executable not found.

      pg_config is required to build psycopg2 from source.  Please add the directory
      containing pg_config to the $PATH or specify the full executable path with the
      option:

          python setup.py build_ext --pg-config /path/to/pg_config build ...

      or with the pg_config option in 'setup.cfg'.

      If you prefer to avoid building psycopg2 from source, please install the PyPI
      'psycopg2-binary' package instead.

      For further information please check the 'doc/src/install.rst' file (also at
      <https://www.psycopg.org/docs/install.html>).

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

Ã Getting requirements to build wheel did not run successfully.
â exit code: 1
â°â> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
dbt -v
./dbt-forklift.sh: line 61: dbt: command not found
./dbt-forklift.sh: line 62: dbt: command not found

davidwitk commented 3 weeks ago

@dbeatty10 Do you know if this only relevant for upgrading to 1.8? Or is this going to affect future versions, too? Context: I wonder if I should add the workaround as general recommendation for upgrading dbt

ann8ty commented 2 weeks ago

I also have this issue on upgrade.

python --version Python 3.11.9 I use venv with vs code

dbt --version Traceback (most recent call last): File "/Users/ann/code/data/data-workflows/.venv/bin/dbt", line 5, in from dbt.cli.main import cli File "/Users/ann/code/data/data-workflows/.venv/lib/python3.11/site-packages/dbt/cli/init.py", line 1, in from .main import cli as dbt_cli # noqa ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ann/code/data/data-workflows/.venv/lib/python3.11/site-packages/dbt/cli/main.py", line 14, in from dbt.cli import requires, params as p File "/Users/ann/code/data/data-workflows/.venv/lib/python3.11/site-packages/dbt/cli/requires.py", line 10, in from dbt.adapters.factory import adapter_management, register_adapter, get_adapter ModuleNotFoundError: No module named 'dbt.adapters.factory'

pip freeze | grep 'dbt' airflow-dbt-python==2.0.1 dbt-adapters==1.3.2 dbt-common==1.5.0 dbt-core==1.8.3 dbt-extractor==0.5.1 dbt-postgres==1.7.17 dbt-redshift==1.7.7 dbt-semantic-interfaces==0.5.1 dbt-tests-adapter==1.9.1

pip install --force-reinstall dbt-adapters

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. dbt-postgres 1.7.17 requires dbt-core==1.7.17, but you have dbt-core 1.8.3 which is incompatible. dbt-redshift 1.7.7 requires dbt-core~=1.7.0, but you have dbt-core 1.8.3 which is incompatible.

my req file airflow-dbt-python==2.0.1 # latest Jul 2024 dbt-core==1.7.17 # airflow-dbt-python 2.0.1 depends on dbt-core<1.8.0 and >=1.4.0; python_version >= "3.11" dbt-redshift==1.7.7

my test req file dbt-tests-adapter

but dbt works (But wrong version)

dbt --version
Core:

installed: 1.8.3
latest: 1.8.3 - Up to date!

Plugins:

was concerned that dbt had moved from 1.7.17 to 1.8.3 when it was pinned, changed my dbt tests file

dbt-tests-adapter==1.7.17

pip freeze | grep 'dbt' airflow-dbt-python==2.0.1 dbt-adapters==1.3.2 dbt-common==1.5.0 dbt-core==1.7.17 dbt-extractor==0.5.1 dbt-postgres==1.7.17 dbt-redshift==1.7.7 dbt-semantic-interfaces==0.4.4 dbt-tests-adapter==1.7.17

dbt --version Core:

installed: 1.7.17
latest: 1.8.3 - Update available!

Your version of dbt-core is out of date! You can find instructions for upgrading here: https://docs.getdbt.com/docs/installation

Plugins:

redshift: 1.7.7 - Update available!
postgres: 1.7.17 - Update available!

At least one plugin is out of date or incompatible with dbt-core. You can find instructions for upgrading here: https://docs.getdbt.com/docs/installation

dbeatty10 commented 2 weeks ago

@dbeatty10 Do you know if this only relevant for upgrading to 1.8? Or is this going to affect future versions, too? Context: I wonder if I should add the workaround as general recommendation for upgrading dbt

@davidwitk I'm assuming this will apply anytime upgrading dbt <1.8.0 to dbt >= 1.8.0.

dbeatty10 commented 2 weeks ago

@ann8ty I'd recommend trying the commands in https://github.com/dbt-labs/dbt-core/issues/10135#issuecomment-2113728550.

brabster commented 2 weeks ago

Can you run 1.7.x adapter implementations with dbt 1.8.x core/adapters? If not then that might explain @ann8ty problem, looks like they have pinned exact 1.7.17 Pg/Redshift adapter versions.

More general question maybe aimed at @dbeatty10 - If you folks are decoupling versions of adapter implementations from core and adapters packages, how can a user tell whether different versions are compatible? Probably some docs somewhere I haven't found?

smittal086 commented 1 week ago

pip install --upgrade dbt-adapters

run this command, hopefully it will solve the issue

jessanmen1 commented 16 hours ago

pip install -r requirements.txt --no-cache-dir --ignore-installed worked for me on Python 3.10.

dbt-labs / dbt-core