Closed HenrikPoulsen closed 2 years ago
matteius
commented
2 years ago I agree with this report and was looking for someone with the same report -- our lock file is 75% lighter after version 2021.5.29. I am currently diffing the code base to see if I can find any hints at what could be going on.
matteius
commented
2 years ago However @HenrikPoulsen, I am still able to reproduce the issue without having the [[source]] section in the Pipfile so I think that is maybe unrelated. Unfortunately my example is a private Pipfile that I cannot share, but it seems to depend on the versions of things in the Pipfile. For example, after modifying just the version of 1/65 requirements and adding another requirement to the Pipfile, the lock file goes from missing 75% of requirements to appearing to be complete.
jfly
commented
2 years ago We're running into this as well. I've put together a repro here: https://github.com/jfly/pipenv-issue-4885-repro. Unlike @matteius but like @HenrikPoulsen, I needed to introduce an alternative Python package index to trigger the issue. So maybe @matteius is running into something different, or there are other ways to get this issue to rear its head.
I'll try to get a clone of pipenv running and see if I can git bisect my way to a problematic commit.
jfly
commented
2 years ago So I bisected, and here's what I found:
That corresponds to this massive PR from @frostming: https://github.com/pypa/pipenv/pull/4759/, which definitely sounds suspicious. I'm not sure how much further I'll be able to take this investigation.
jfly
commented
2 years ago Ok, I've dug enough to confirm that 5ef5a59d41385945f2fc7f0e492900921ffc924b is definitely where this broke.
Previous, working behavior (I'm looking at d0fa43ad973d5c427f0e2baeee37de577651725a):
actually_resolve_deps calls resolver.resolve_hashes: https://github.com/pypa/pipenv/blob/d0fa43ad973d5c427f0e2baeee37de577651725a/pipenv/utils.py#L1103Resolver::resolve_hashes calls self.resolver.resolve_hashes: https://github.com/pypa/pipenv/blob/d0fa43ad973d5c427f0e2baeee37de577651725a/pipenv/utils.py#L977Resolver::resolve_hashes finds all the hashes: https://github.com/pypa/pipenv/blob/d0fa43ad973d5c427f0e2baeee37de577651725a/pipenv/patched/piptools/resolver.py#L142-L149New, broken behavior (I'm looking at 3ab4763d4ac203fdc2bd16e9609bebe311744bc1 here):
actually_resolve_deps calls resolver.resolve_hashes: https://github.com/pypa/pipenv/blob/3ab4763d4ac203fdc2bd16e9609bebe311744bc1/pipenv/utils.py#L1090Resolver::resolve_hashes calls self.collect_hashes: https://github.com/pypa/pipenv/blob/3ab4763d4ac203fdc2bd16e9609bebe311744bc1/pipenv/utils.py#L964
3a. When using pypi.org collect_hashes goes into this pypi.org specific code block and calls _get_hashes_from_pypi, which finds all the hashes, regardless of the platform we're running on: https://github.com/pypa/pipenv/blob/3ab4763d4ac203fdc2bd16e9609bebe311744bc1/pipenv/utils.py#L945-L951
3b. When using a custom python package index collect_hashes calls self.finder.find_best_candidate, which only finds candidates that are installable on the current platform: https://github.com/pypa/pipenv/blob/3ab4763d4ac203fdc2bd16e9609bebe311744bc1/pipenv/utils.py#L953-L959Unfortunately, this seems to be one of those situations where understanding why it used to work and why it doesn't work now doesn't seem to be enough information to fix the problem. I don't know why we moved away from piptools, but presumably that was for a good reason. Maybe the right fix is to change up this code to find all candidates, not just ones that could work on the current platform?
@frostming, I could really use some help here. Thanks!
frostming
commented
2 years ago When using a custom python package index collect_hashes calls self.finder.find_best_candidate, which only finds candidates that are installable on the current platform.
This should be wrong, care to submit a fix for it? @jfly Thanks.
jfly
commented
2 years ago @frostming, I can give that a shot. I could use a little help getting started, though. My initial thought is to add a boolean parameter to find_best_candidate that tells it it's ok to find candidates that for other platforms (perhaps a only_for_current_platform parameter that defaults to true, and we can change it to False when collect_hashes runs). How does that sound?
jfly
commented
2 years ago Oh wait, I just realized I'm talking about changing vendored pip code. Is that an ok? Or do we prefer not to change that code?
matteius
commented
2 years ago @jfly This other issue #4866 was specifically what was causing the missing hashes in my case; Based on your research so far do you think this could be related to what you have found? https://github.com/pypa/pipenv/issues/4866#issuecomment-968729805
jfly
commented
2 years ago @matteius that sounds different. The bug I tracked down is definitely specific to when you're using a custom python package server.
Issue description
Since some version after pipenv 2021.5.29 it seems the lockfile behavior has changed regarding the hashes that get written to the file. I use Renovate to create PRs for updated dependencies. And since a couple of weeks the repo started to get lockfiles produced that were missing windows hashes for dependencies. Specifically it updates from aiohttp 3.8.0 to 3.8.1 and it ends up only including two hashes for it in the lockfile after it is done. When CI then tries to run the windows unit tests on that PR it fails, due to the hashes it is looking for is not present in the lockfile.
I have noticed that this only happens if I have a custom index-url in the Pipfile, which in my case points to an artifactory instance we use.
Expected result
Lockfile should be updated with all available hashes so crossplatform
pipenv installis successfulActual result
I get this error message on windows after it pip installs using the lockfile that renovate has created:
It only includes 2 hashes instead of the 70+ as on the older pipenv version This is the verbose output of when it doesn't work:
Steps to replicate
pipenv --supportbelow here, modify the index-url to something that works for youdocker run -it -v /folder/with/pipfile:/data --entrypoint /bin/bash renovate/python:3.9use whatever you preferpip install pipenvpip installPipfile.lockonly has a couple of values in itObserve that I was unable to reproduce this without the
[[source]]section in the Pipfile$ pipenv --support
Pipenv version: `'2021.11.23'` Pipenv location: `'/home/ubuntu/.local/lib/python3.9/site-packages/pipenv'` Python location: `'/usr/local/python/3.9.9/bin/python3.9'` Python installations found: - `3.9.9`: `/usr/local/python/3.9.9/bin/python3.9` - `3.9.9`: `/usr/local/python/3.9.9/bin/python3` - `3.9.9`: `/usr/local/python/3.9.9/bin/python` - `3.9.9`: `/usr/local/bin/python` PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.9.9', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_python_implementation': 'CPython', 'platform_release': '5.10.60.1-microsoft-standard-WSL2', 'platform_system': 'Linux', 'platform_version': '#1 SMP Wed Aug 25 23:20:18 UTC 2021', 'python_full_version': '3.9.9', 'python_version': '3.9', 'sys_platform': 'linux'} ``` System environment variables: - `USER_ID` - `HOSTNAME` - `PWD` - `USER_NAME` - `HOME` - `USER_HOME` - `LANG` - `LS_COLORS` - `TERM` - `SHLVL` - `BASH_ENV` - `LC_ALL` - `PATH` - `DEBIAN_FRONTEND` - `BUILDPACK` - `OLDPWD` - `_` - `PIP_SHIMS_BASE_MODULE` - `PIP_DISABLE_PIP_VERSION_CHECK` - `PYTHONDONTWRITEBYTECODE` - `PIP_PYTHON_PATH` - `PYTHONFINDER_IGNORE_UNSUPPORTED` Pipenv–specific environment variables: Debug–specific environment variables: - `PATH`: `/home/ubuntu/.local/bin:/usr/local/python/3.9.9/bin:/home/ubuntu/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin` - `LANG`: `C.UTF-8` - `PWD`: `/data` --------------------------- Contents of `Pipfile` ('/data/Pipfile'): ```toml [[source]] url = "https//someartifactory/api/pypi/pypi/simple" verify_ssl = true name = "pypi" [packages] aiohttp = "==3.8.1" ``` Contents of `Pipfile.lock` ('/data/Pipfile.lock'): ```json { "_meta": { "hash": { "sha256": "124cf509d4d27ce063e7a4f1fe5faf9892b0896c8e84f4207b11663cc780b28e" }, "pipfile-spec": 6, "requires": {}, "sources": [ { "name": "pypi", "url": "https//someartifactory/api/pypi/pypi/simple", "verify_ssl": true } ] }, "default": { "aiohttp": { "hashes": [ "sha256:d15367ce87c8e9e09b0f989bfd72dc641bcd04ba091c68cd305312d00962addd", "sha256:fc5471e1a54de15ef71c1bc6ebe80d4dc681ea600e68bfd1cbce40427f0b7578" ], "index": "pypi", "version": "==3.8.1" }, "aiosignal": { "hashes": [ "sha256:26e62109036cd181df6e6ad646f91f0dcfd05fe16d0cb924138ff2ab75d64e3a", "sha256:78ed67db6c7b7ced4f98e495e572106d5c432a93e1ddd1bf475e1dc05f5b7df2" ], "markers": "python_version >= '3.6'", "version": "==1.2.0" }, "async-timeout": { "hashes": [ "sha256:a22c0b311af23337eb05fcf05a8b51c3ea53729d46fb5460af62bee033cec690", "sha256:b930cb161a39042f9222f6efb7301399c87eeab394727ec5437924a36d6eef51" ], "markers": "python_version >= '3.6'", "version": "==4.0.1" }, "attrs": { "hashes": [ "sha256:149e90d6d8ac20db7a955ad60cf0e6881a3f20d37096140088356da6c716b0b1", "sha256:ef6aaac3ca6cd92904cdd0d83f629a15f18053ec84e6432106f7a4d04ae4f5fb" ], "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'", "version": "==21.2.0" }, "charset-normalizer": { "hashes": [ "sha256:735e240d9a8506778cd7a453d97e817e536bb1fc29f4f6961ce297b9c7a917b0", "sha256:83fcdeb225499d6344c8f7f34684c2981270beacc32ede2e669e94f7fa544405" ], "markers": "python_full_version >= '3.5.0'", "version": "==2.0.8" }, "frozenlist": { "hashes": [ "sha256:1fef737fd1388f9b93bba8808c5f63058113c10f4e3c0763ced68431773f72f9", "sha256:68201be60ac56aff972dc18085800b6ee07973c49103a8aba669dee3d71079de" ], "markers": "python_version >= '3.6'", "version": "==1.2.0" }, "idna": { "hashes": [ "sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff", "sha256:9d643ff0a55b762d5cdb124b8eaa99c66322e2157b69160bc32796e824360e6d" ], "markers": "python_full_version >= '3.5.0'", "version": "==3.3" }, "multidict": { "hashes": [ "sha256:0dd1c93edb444b33ba2274b66f63def8a327d607c6c790772f448a53b6ea59ce", "sha256:93de39267c4c676c9ebb2057e98a8138bade0d806aad4d864322eee0803140a0" ], "markers": "python_version >= '3.6'", "version": "==5.2.0" }, "typing-extensions": { "hashes": [ "sha256:2cdf80e4e04866a9b3689a51869016d36db0814d84b8d8a568d22781d45d27ed", "sha256:829704698b22e13ec9eaf959122315eabb370b0884400e9818334d8b677023d9" ], "markers": "python_version >= '3.6'", "version": "==4.0.0" }, "yarl": { "hashes": [ "sha256:45399b46d60c253327a460e99856752009fcee5f5d3c80b2f7c0cae1c38d56dd", "sha256:f64394bd7ceef1237cc604b5a89bf748c95982a84bcd3c4bbeb40f685c810794" ], "markers": "python_version >= '3.6'", "version": "==1.7.2" } }, "develop": {} } ```