[CT-2135] [Bug] Dbt-snowflake dependency upper bounds do not allow for pointing at 1.5.0a1 revision

[fadi-circleci]

Is this a new bug in dbt-core?

• [X] I believe this is a new bug in dbt-core
• [X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

In order to resolve some security vulnerabilities we have pointed to this commit https://github.com/dbt-labs/dbt-snowflake/commit/ce23a989cbc4eff7bdb8e438b8c3eff9415b9ea2 as our version in our pyproject.toml.

In order to resolve this dependency poetry attempts to install dbt-core >=1.5.0a1,<1.6.0 which it can not find.

Because dbt-snowflake (1.5.0a1) @ git+https://github.com/dbt-labs/dbt-snowflake@ce23a989cbc4eff7bdb8e438b8c3eff9415b9ea2 depends on dbt-core (>=1.5.0a1,<1.6.0) which doesn't match any versions, dbt-snowflake is forbidden. So, because dbt-data-modeling depends on dbt-snowflake (1.5.0a1) @ git+https://github.com/dbt-labs/dbt-snowflake@ce23a989cbc4eff7bdb8e438b8c3eff9415b9ea2, version solving failed.

Expected Behavior

Expected behavior is that 1.4.x would allow flexibility on which version of dbt-snowflake it can use OR wheels be made available for 1.5.0a1. The upgrade here is needed in order to patch a security vulnerability in the cyrptography package.

Steps To Reproduce

1. The enviornment is listed below
2. pyproject.toml `[build-system] build-backend = "poetry.masonry.api" requires = ["poetry>=0.12"]

[tool.poetry] description = "Where the Data Platform maintained core data models live." name = "dbt_data_modeling" version = "0.0.1"

[tool.poetry.scripts] model = "dbt_data_modeling.main:program.run"

[tool.poetry.dependencies] datadog = "^0.39"

magicinvoke = "^2.4" python = "^3.8" rollbar = "^0.15" boto3 = "^1.20" snowflake-connector-python = {version = "*", extras = ["secure-local-storage"]} dbt-snowflake = {git = "https://github.com/dbt-labs/dbt-snowflake", rev = "ce23a989cbc4eff7bdb8e438b8c3eff9415b9ea2"} typed-ast = "^1.5" cryptography = "^39.0.0" dbt-core = "^1.4.1" lkml = "^1.3.1" looker-sdk = "^22.20.0" re-data = "^0.10.7"`

3. Attempt to advance cryptography package to 39.0 must advance dbt-snowflake which is out of bounds for dbt-core 1.4.x
4. see error below

Relevant log output

Attempt to install a specific commit. 

➜  resources git:(test_dbt_version) ✗ poetry add git+https://github.com/dbt-labs/dbt-core@d74ae19523122d63b0281e176158a6851204a711      

Unable to determine package info for path: /Users/fadi/Library/Caches/pypoetry/virtualenvs/dbt-data-modeling-6_5Rm1wY-py3.10/src/dbt-core

Command ['/var/folders/0p/w8xvql850cd0sw4jvwq8y53m0000gn/T/tmp960b1h16/.venv/bin/python', '-'] errored with the following return code 1, and output: 
Traceback (most recent call last):
  File "/private/var/folders/0p/w8xvql850cd0sw4jvwq8y53m0000gn/T/tmp960b1h16/.venv/lib/python3.10/site-packages/build/__init__.py", line 422, in _handle_backend
    yield
  File "/private/var/folders/0p/w8xvql850cd0sw4jvwq8y53m0000gn/T/tmp960b1h16/.venv/lib/python3.10/site-packages/build/__init__.py", line 310, in get_requires_for_build
    return set(get_requires(config_settings))
  File "/private/var/folders/0p/w8xvql850cd0sw4jvwq8y53m0000gn/T/tmp960b1h16/.venv/lib/python3.10/site-packages/pep517/wrappers.py", line 172, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
  File "/private/var/folders/0p/w8xvql850cd0sw4jvwq8y53m0000gn/T/tmp960b1h16/.venv/lib/python3.10/site-packages/pep517/wrappers.py", line 322, in _call_hook
    self._subprocess_runner(
  File "/private/var/folders/0p/w8xvql850cd0sw4jvwq8y53m0000gn/T/tmp960b1h16/.venv/lib/python3.10/site-packages/build/__init__.py", line 261, in _runner
    self._hook_runner(cmd, cwd, extra_environ)
  File "/private/var/folders/0p/w8xvql850cd0sw4jvwq8y53m0000gn/T/tmp960b1h16/.venv/lib/python3.10/site-packages/pep517/wrappers.py", line 75, in quiet_subprocess_runner
    check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)
  File "/usr/local/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/var/folders/0p/w8xvql850cd0sw4jvwq8y53m0000gn/T/build-env-qr8qs5ai/bin/python', '/private/var/folders/0p/w8xvql850cd0sw4jvwq8y53m0000gn/T/tmp960b1h16/.venv/lib/python3.10/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/var/folders/0p/w8xvql850cd0sw4jvwq8y53m0000gn/T/tmpsjozfln7']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 16, in <module>
  File "/private/var/folders/0p/w8xvql850cd0sw4jvwq8y53m0000gn/T/tmp960b1h16/.venv/lib/python3.10/site-packages/build/__init__.py", line 309, in get_requires_for_build
    with self._handle_backend(hook_name):
  File "/usr/local/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/private/var/folders/0p/w8xvql850cd0sw4jvwq8y53m0000gn/T/tmp960b1h16/.venv/lib/python3.10/site-packages/build/__init__.py", line 430, in _handle_backend
    raise BuildBackendException(exception, f'Backend subproccess exited when trying to invoke {hook}')
build.BuildBackendException: Backend subproccess exited when trying to invoke get_requires_for_build_wheel
input was : import build
import build.env
import pep517

source = '/Users/fadi/Library/Caches/pypoetry/virtualenvs/dbt-data-modeling-6_5Rm1wY-py3.10/src/dbt-core'
dest = '/var/folders/0p/w8xvql850cd0sw4jvwq8y53m0000gn/T/tmp960b1h16/dist'

with build.env.IsolatedEnvBuilder() as env:
    builder = build.ProjectBuilder(
        srcdir=source,
        scripts_dir=env.scripts_dir,
        python_executable=env.executable,
        runner=pep517.quiet_subprocess_runner,
    )
    env.install(builder.build_system_requires)
    env.install(builder.get_requires_for_build('wheel'))
    builder.metadata_path(dest)

No fallback setup.py file was found to generate egg_info.

Environment

System information:
 Kernel Version: 5.15.0-1021-aws
 Operating System: Ubuntu 20.04.5 LTS
 OSType: linux
 Architecture: x86_64
 python:3.9.16 and 3.10latest stable prob
 poetry: 1.2.2
 dbt-core: 1.4.x

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

[dbeatty10]

Hey @fadi-circleci !

This is actually behaving as expected -- we intend for dbt-snowflake==1.5.0a1 to require dbt-core>=1.5.0a1,<1.6.0.

So to install dbt-snowflake from hash ce23a989cbc4eff7bdb8e438b8c3eff9415b9ea2, you'd also need to install a compatible hash from dbt-core (maybe c952d44ec5c2506995fbad75320acbae49125d3d? 🤷 )

More generally, dbt-snowflake 1.x requires dbt-core 1.x also, and the same goes for all the adapters we maintain (dbt-postgres, dbt-redshfit, dbt-snowflake, and dbt-spark).

The actual solution

We are planning on releasing a 1.4.x release that will include the cryptography>= 39.0.0 that you are seeking, but I don't know the date for it yet.

In the meantime, there's a multiple different ideas you could try, but they are all a bit tricky.

One idea, if you can't wait

Install dbt-snowflake==1.4.1 & force upgrade to cryptography>= 39.0.0 as a post step. I didn't try the following at all, but maybe something like this 🤷:

poetry install
poetry shell
python -m pip install cryptography>=39.0.0

Another idea, if you can't wait

Fork dbt-snowflake==1.4.1 and patch setup.py here. Then install your fork.

Wrapping up

While I understand that this might not meet your short-term needs, the only option we can explicitly support is for you to wait for us to publish dbt-snowflake==1.4.2. Everything else is just a creative idea that should not be interpreted as a recommendation.

Since we have a patch release of dbt-snowflake planned, and we aren't planning to change our approach to version compatibility between dbt-snowflake and dbt-core, I'm going to close this as wontfix.