search

dependabot / dependabot-core

🤖 Dependabot's core logic for creating update PRs.
https://docs.github.com/en/code-security/dependabot
MIT License
4.66k stars 1.01k forks source link

Dependabot fails to look up deps with extras in pyproject.toml #2855

Open halfdan opened 3 years ago

halfdan commented 3 years ago

Package manager/ecosystem python:poetry

Manifest contents prior to update

[tool]
[tool.poetry]
authors = ["halfdan <halfdan@valhalla.is>"]
description = ""
name = "koios"
version = "0.0.0"

[tool.poetry.dependencies]
attrs = "*"
boto3 = "==1.15.16"
click = "*"
colorama = "==0.4.1"
connexion = {extras = ["swagger-ui"], version = "*"}
datadog = "*"
ddtrace = "*"
flask = "*"
flask-log-request-id = "*"
flask-sqlalchemy = "*"
gevent = "*"
gunicorn = {extras = ["gevent"], version = "*"}
jinja2 = "*"
numpy = "*"
pandas = "<1.0.0"
prance = "*"
psycogreen = "*"
psycopg2 = "*"
pyhive = {extras = ["presto"], version = "*"}
python = "==3.8.6"
python-json-logger = "*"
simplejson = "*"
smart-open = "*"
sqlalchemy = "*"
sqlalchemy-redshift = "*"
structlog = "*"
xlsxwriter = "*"

[tool.poetry.dev-dependencies]
bandit = "*"
black = "==19.10b0"
flake8 = "*"
freezegun = "*"
ipynb = "*"
ipython = "*"
jupyter = "==1.0.0"
mccabe = "*"
moto = "==1.3.14"
nbformat = "*"
openapi-spec-validator = "*"
pyparsing = "*"
pytest = "*"
pytest-cov = "*"
pytest-timeout = "*"
xlrd = "*"
yamllint = "^1.25.0"

Updated dependency

N/A

What you expected to see, versus what you actually saw

I expect dependabot to be able to update dependencies and create PRs.

Additional info: We have migrated from Pipenv. We are using a custom python registry (gemfury). The configured registry worked fine w/ Dependabot when we were using Pipenv and wasn't changed after migrating to poetry. Local installs are working.

Images of the diff or a link to the PR, issue or logs

[...]
  proxy | 2020/12/14 06:31:04 [104] 404 https://pypi.fury.io:443/ourcompany/connexion/
updater | INFO <job_59084963> Handled error whilst updating moto: dependency_file_not_resolvable {:message=>"Creating virtualenv koios--G9gDakr-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs\nUpdating dependencies\nResolving dependencies...\n\n  SolverProblemError\n\n  Because koios depends on pyhive[presto] (*) which doesn't match any versions, version solving failed.\n\n  at /usr/local/.pyenv/versions/3.8.6/lib/python3.8/site-packages/poetry/puzzle/solver.py:241 in _solve\n      237│             packages = result.packages\n      238│         except OverrideNeeded as e:\n      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)\n      240│         except SolveFailure as e:\n    → 241│             raise SolverProblemError(e)\n      242│ \n      243│         results = dict(\n      244│             depth_first_search(\n      245│                 PackageNode(self._package, packages), aggregate_package_nodes\n"}
updater | INFO <job_59084963> Finished job processing
Creating virtualenv koios-Q1MVnhBj-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

  SolverProblemError

  Because koios depends on pyhive[presto] (*) which doesn't match any versions, version solving failed.

  at /usr/local/.pyenv/versions/3.8.6/lib/python3.8/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes
ulgens commented 3 years ago

Did you add private repository information to settings?

image

Under https://app.dependabot.com/accounts/<your-organization>/config-variables I encounted the same issue last week and that change solved it for our repo.

halfdan commented 3 years ago

@ulgens, yes I did and it was working when the same repo used Pipfile/Pipfile.lock with pipenv. I only changed the dependency manager.

lucaskjaero commented 3 years ago

I am also affected by this. Would it be possible to reproduce the issue using poetry commands? Or is this specific to dependabot.

michakinchen1988 commented 3 years ago

I don't know what ya are doing but I bought a brand new phone within 10 min I had linieux in it and dependabot core working I've had dependabot complete since 2008 I just learned recently figuring out how to certified it the merge is being block by security features I got the Mac os to in phone but it makes the security for the browser not responding microsoft enderpoint

michakinchen1988 commented 3 years ago

Package manager/ecosystem python:poetry

Manifest contents prior to update

[tool]
[tool.poetry]
authors = ["halfdan <halfdan@valhalla.is>"]
description = ""
name = "koios"
version = "0.0.0"

[tool.poetry.dependencies]
attrs = "*"
boto3 = "==1.15.16"
click = "*"
colorama = "==0.4.1"
connexion = {extras = ["swagger-ui"], version = "*"}
datadog = "*"
ddtrace = "*"
flask = "*"
flask-log-request-id = "*"
flask-sqlalchemy = "*"
gevent = "*"
gunicorn = {extras = ["gevent"], version = "*"}
jinja2 = "*"
numpy = "*"
pandas = "<1.0.0"
prance = "*"
psycogreen = "*"
psycopg2 = "*"
pyhive = {extras = ["presto"], version = "*"}
python = "==3.8.6"
python-json-logger = "*"
simplejson = "*"
smart-open = "*"
sqlalchemy = "*"
sqlalchemy-redshift = "*"
structlog = "*"
xlsxwriter = "*"

[tool.poetry.dev-dependencies]
bandit = "*"
black = "==19.10b0"
flake8 = "*"
freezegun = "*"
ipynb = "*"
ipython = "*"
jupyter = "==1.0.0"
mccabe = "*"
moto = "==1.3.14"
nbformat = "*"
openapi-spec-validator = "*"
pyparsing = "*"
pytest = "*"
pytest-cov = "*"
pytest-timeout = "*"
xlrd = "*"
yamllint = "^1.25.0"

Updated dependency

N/A

What you expected to see, versus what you actually saw

I expect dependabot to be able to update dependencies and create PRs.

Additional info: We have migrated from Pipenv. We are using a custom python registry (gemfury). The configured registry worked fine w/ Dependabot when we were using Pipenv and wasn't changed after migrating to poetry. Local installs are working.

Images of the diff or a link to the PR, issue or logs

[...]
  proxy | 2020/12/14 06:31:04 [104] 404 https://pypi.fury.io:443/ourcompany/connexion/
updater | INFO <job_59084963> Handled error whilst updating moto: dependency_file_not_resolvable {:message=>"Creating virtualenv koios--G9gDakr-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs\nUpdating dependencies\nResolving dependencies...\n\n  SolverProblemError\n\n  Because koios depends on pyhive[presto] (*) which doesn't match any versions, version solving failed.\n\n  at /usr/local/.pyenv/versions/3.8.6/lib/python3.8/site-packages/poetry/puzzle/solver.py:241 in _solve\n      237│             packages = result.packages\n      238│         except OverrideNeeded as e:\n      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)\n      240│         except SolveFailure as e:\n    → 241│             raise SolverProblemError(e)\n      242│ \n      243│         results = dict(\n      244│             depth_first_search(\n      245│                 PackageNode(self._package, packages), aggregate_package_nodes\n"}
updater | INFO <job_59084963> Finished job processing
Creating virtualenv koios-Q1MVnhBj-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

  SolverProblemError

  Because koios depends on pyhive[presto] (*) which doesn't match any versions, version solving failed.

  at /usr/local/.pyenv/versions/3.8.6/lib/python3.8/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

[tool] [tool.poetry] authors = ["halfdan halfdan@valhalla.is"] description = "" name = "koios" version = "0.0.0"

[tool.poetry.dependencies] attrs = "" boto3 = "==1.15.16" click = "" colorama = "==0.4.1" connexion = {extras = ["swagger-ui"], version = ""} datadog = "" ddtrace = "" flask = "" flask-log-request-id = "" flask-sqlalchemy = "" gevent = "" gunicorn = {extras = ["gevent"], version = ""} jinja2 = "" numpy = "" pandas = "<1.0.0" prance = "" psycogreen = "" psycopg2 = "" pyhive = {extras = ["presto"], version = ""} python = "==3.8.6" python-json-logger = "" simplejson = "" smart-open = "" sqlalchemy = "" sqlalchemy-redshift = "" structlog = "" xlsxwriter = "*"

[tool.poetry.dev-dependencies] bandit = "" black = "==19.10b0" flake8 = "" freezegun = "" ipynb = "" ipython = "" jupyter = "==1.0.0" mccabe = "" moto = "==1.3.14" nbformat = "" openapi-spec-validator = "" pyparsing = "" pytest = "" pytest-cov = "" pytest-timeout = "" xlrd = "*" yamllint = "^1.25.0"

cancan101 commented 2 years ago

I believe this issue also applies to requirements.txt files, namely that dependabot is not putting up PRs to bump the versions of any packages in that specify extras. For example, I have this in my file: uvicorn[standard]==0.15.0 however the newest version is 0.16.0, but no PR has been put up for this new version.

jeffwidman commented 1 year ago

Notes to whoever next looks at this... the important bits to look at are:

# pyproject.toml
pyhive = {extras = ["presto"], version = "*"}

results in error log:

Because koios depends on pyhive[presto] (*) which doesn't match any versions, version solving failed.

So the version resolving is failing for some reason...