search

dependabot / dependabot-core

🤖 Dependabot's core logic for creating update PRs.
https://docs.github.com/en/code-security/dependabot
MIT License
4.63k stars 991 forks source link

Dependabot couldn't fetch all your path-based dependencies #8428

Closed HangJung97 closed 6 months ago

HangJung97 commented 10 months ago

Is there an existing issue for this?

Package ecosystem

pip

Package manager version

No response

Language version

Python

Manifest location and content before the Dependabot update

https://github.com/HangJung97/test_dependabot/blob/main/pyproject.toml

dependabot.yml content

version: 2
updates:
  - package-ecosystem: "pip" # See documentation for possible values
    directory: "/" # Location of package manifests
    schedule:
      interval: "daily"
    allow:
      - dependency-type: "production"
    ignore:
      - dependency-name: "pytorch-lightning"
        update-types: ["version-update:semver-patch"]
      - dependency-name: "torchmetrics"
        update-types: ["version-update:semver-patch"]
  - package-ecosystem: "gitsubmodule"
    directory: "/"
    schedule:
      interval: "monthly"
    allow:
      - dependency-type: "production"

Updated dependency

No response

What you expected to see, versus what you actually saw

I have a git submodule named ASCENT located at the root. I have included it as a package in my pyproject.toml like this ascent = { path = "./ASCENT/", develop = true }. I expected dependabot to be able to find the pyproject.toml in my ASCENT submodule since my path is relative and correct but got this error instead Dependabot couldn't fetch all your path-based dependencies. The exact error looks like this:

updater | 2023/11/17 18:56:10 ERROR Error during file fetching; aborting: The following path based dependencies could not be retrieved: ASCENT/pyproject.toml.

The public repository I created to debug this problem can be found here.

Native package manager behavior

No response

Images of the diff or a link to the PR, issue, or logs

No response

Smallest manifest that reproduces the issue

No response

deivid-rodriguez commented 10 months ago

Hi!

We recently started cloning full python repositories when running updates, however we don't yet clone submodules.

We'd need to add this method to the python fetcher, like we do for Javascript.

Would you be interested in creating a PR to fix this?

HangJung97 commented 9 months ago

Hi,

I'm afraid that I won't have time to add the recurse_submodules_when_cloning method. Besides, I'm not that familiar with Ruby language. If you can provide me with an example (e.g., what is done for Javascript), I could take a look at that, but I can't guarantee anything.

Best, Hang Jung

deivid-rodriguez commented 9 months ago

No problem.

This is what the commit enabling it for JavaScript looked like: https://github.com/dependabot/dependabot-core/pull/6718/commits/e2eec88329f73b4925e9b77c10de0ad37d711647.

Then you could use the CLI or dry-run.rb script as explained in our README to verify that the problem is fixed in the demo repository that you created.

HangJung97 commented 9 months ago

@deivid-rodriguez, unfortunately, I won't have time to look into this issue. I'm really sorry for that. Do you mind fixing this issue?

deivid-rodriguez commented 9 months ago

No problem, we will eventually prioritize and fix this issue. For now, I'll set the "good first issue" label here, since someone from the community stepping in would certainly speed this up!

ReenigneArcher commented 8 months ago

I'm facing the same issue after adding a python dependency as a submodule. They are no longer publishing PyPI packages, so I figured I could get dependabot to update it if I added it as a submodule.

Dependabot couldn't fetch all your path-based dependencies
The affected dependencies were "./third-party/youtube-dl" at /requirements.txt.

To use path-based dependencies with Dependabot the paths must be relative, resolve to a directory in this project's source code, and contain a valid Python project.
  proxy | 2023/12/27 10:24:35 proxy starting, commit: 02a8910b917eff32ef3fe812e35a131d6286bc20
  proxy | 2023/12/27 10:24:35 Listening (:1080)
updater | 2023-12-27T10:24:37.700734163 [766916747:main:WARN:src/devices/src/legacy/serial.rs:222] Detached the serial input due to peer close/error.
updater | time="2023-12-27T10:24:42Z" level=info msg="guest starting" commit=eb5aa56302357f07a0e790713fa099f11a1af831
updater | time="2023-12-27T10:24:42Z" level=info msg="starting job..." fetcher_timeout=10m0s job_id=766916747 updater_timeout=45m0s updater_version=f326cfd6c730e8440795643c75f61de0a85634c4-pip
updater | 2023/12/27 10:24:49 INFO Raven 3.1.2 ready to catch errors
updater | 2023/12/27 10:24:54 INFO <job_766916747> Starting job processing
  proxy | 2023/12/27 10:24:55 [002] GET https://github.com:443/LizardByte/Themerr-plex/info/refs?service=git-upload-pack
  proxy | 2023/12/27 10:24:55 [002] * authenticating git server request (host: github.com)
  proxy | 2023/12/27 10:24:55 [002] 200 https://github.com:443/LizardByte/Themerr-plex/info/refs?service=git-upload-pack
  proxy | 2023/12/27 10:24:55 [004] POST https://github.com:443/LizardByte/Themerr-plex/git-upload-pack
  proxy | 2023/12/27 10:24:55 [004] * authenticating git server request (host: github.com)
  proxy | 2023/12/27 10:24:55 [004] 200 https://github.com:443/LizardByte/Themerr-plex/git-upload-pack
  proxy | 2023/12/27 10:24:55 [006] POST https://github.com:443/LizardByte/Themerr-plex/git-upload-pack
  proxy | 2023/12/27 10:24:55 [006] * authenticating git server request (host: github.com)
  proxy | 2023/12/27 10:24:55 [006] 200 https://github.com:443/LizardByte/Themerr-plex/git-upload-pack
updater | 2023/12/27 10:24:58 ERROR <job_766916747> Error during file fetching; aborting: The following path based dependencies could not be retrieved: "./third-party/youtube-dl" at /requirements.txt
updater | 2023/12/27 10:24:59 INFO <job_766916747> Finished job processing
updater | 2023/12/27 10:24:59 INFO Results:
updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details.
updater | +---------------------------------+
updater | |             Errors              |
updater | +---------------------------------+
updater | | path_dependencies_not_reachable |
updater | +---------------------------------+
updater | time="2023-12-27T10:24:59Z" level=info msg="task complete" container_id=job-766916747-file-fetcher exit_code=0 job_id=766916747 step=fetcher
updater | time="2023-12-27T10:24:59Z" level=warning msg="failed during fetch, skipping updater" job_id=766916747

Can you just skip path based dependencies in Python instead of erroring out completely? Submodules are already able to be updated by the gitsubmodule ecosystem.