Multi Platform support for Pipfile.lock

ghost commented 2 years ago

Is your feature request related to a problem? Please describe.

Currently it is not possible to create a Pipfile.lock for multiple platforms automatically. Only hashes for the current platform are locked. This is problematic when you want to use Pipefile.lock to share locked dependency versions in a multi platform setup, e.g. development on Windows or Mac and production on Linux.

E.g. cryptography generates the following Pipfile.lock entry on Windows 10 64bit:

        "cryptography": {
            "hashes": [
                "sha256:471e0d70201c069f74c837983189949aa0d24bb2d751b57e26e3761f2f782b8d",
                "sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==37.0.2"
        },

and the following one on a Linux VM:

        "cryptography": {
            "hashes": [
                "sha256:0cc20f655157d4cfc7bada909dc5cc228211b075ba8407c46467f63597c78178",
                "sha256:2bd1096476aaac820426239ab534b636c77d71af66c547b9ddcd76eb9c79e004",
                "sha256:59b281eab51e1b6b6afa525af2bd93c16d49358404f814fe2c2410058623928c",
                "sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==37.0.2"
        },

Describe the solution you'd like

My preferred solution would be a new entry in the Pipfile that defines which platforms hashes must always be included when locking. If the current systems platform isn't in the list it should be be added to the lock list though to retain backwards compatibility.

The Pipfile entry could look like this:

[[source]]
platforms = "manylinux_2_24_x86_64 or win_amd_64"

If possible it would be great if the values could be eagerly validated and an error message printed on invalid values.

Note: Ticket https://github.com/pypa/pipenv/issues/210 is similar to this one, but I was asked to open a new ticket

dijital20 commented 1 year ago

Could the lock file just be per-platform/version/etc?

Pipfile defines my sources and packages.

When I pipenv lock on my Python 3.8 on Windows 64bit, why couldn't it produce a Pipfile.3.8.windows.x64.lock?
If I try to pipenv sync on Python 3.10, since there is not Pipfile.3.10.windows.x64.lock, could the normal behavior kick in and a new lock would be generated?
Likewise, if I pipenv lock on Linux python 3.10, could it generate a Pipfile.3.10.linux.x64.lock?

What would be the drawback to that type of approach, aside from legacy support for the existing Pipfile.lock?

matteius commented 1 year ago

What would the advantage of maintaining multiple lock files per platform and environment @dijital20 ? I am not a big fan of the idea of having platform specific lock files and then how to know which one to use during sync.

could the normal behavior kick in and a new lock would be generated?

Not for sync -- that would be very very bad because you loose the trust that you are verifying the dependencies you used in dev/test are the same you are deploying to production because the hashes would regenerate.

The current design can be extended to account for multiple platforms already -- for multi-platform dependencies all of the associated hashes are included in the lock, so this issue is really about dependencies that are for a different platform than the one you are locking on should also be included in this lock file with the relevant platform marker. Today this only happens if you lock on the environment that requirement is compatible with. Pipenv's own lock file has this problem -- if we do not lock on windows we loose a windows-specific dependency from the lock file.

EDIT: Just re-read the issue report and I am very surprised that the list of hashes for cryptography would be different between those two environments -- this part deserves some additional triage with the latest version of pipenv.

dijital20 commented 1 year ago

Yeah, platform-specific and Python-version-specific dependencies are the issue to solve here.

I had similar issues though when I moved a project from Python 3.8 to Python 3.10. My Pipfile and lock files both had Python 3.8 in them, and the standard pipenv sync --dev failed unless I did pipenv sync --dev --python C:\python310\python.exe, and then re-locked. At that point, everyone using Python 3.8 was out of luck, and so was anyone on Python 3.10.

I am not a big fan of the idea of having platform specific lock files and then how to know which one to use during sync.

pipenv when called, would generate the lock file name based on the interpreter. In my examples, the names were Pipfile.{version}.{platform}.{architecture}.lock. If the file exists, party on. If the file doesn't exist, then it should behave the same way pipenv does today if the lock file doesn't exist.

I am not sure if platform-specific lock files is the solution... just thinking through it as we ran into this today, as a colleague is developing a new Python module that he wants to support both Windows and Linux on, but there are some dependencies which are platform specific.

Some alternatives to consider:

Build the multi platform stuff into the lock file itself, but ideally in a better way than it is now.
Add an option to lock, sync, install, and uninstall to specify a lock file name/path. If not specified, use the default name today. This gives the user option to maintain separate specific lock files for different environments.

matteius commented 1 year ago

@dijital20 or @xyxz-web -- were either of you by chance using private python repositories isntead of pypi? I just locked on the latest pipenv main on windows and cryptography has a lot more hashes -- like all of them -- than the original example had.

        "cryptography": {
            "hashes": [
                "sha256:093cb351031656d3ee2f4fa1be579a8c69c754cf874206be1d4cf3b542042804",
                "sha256:0cc20f655157d4cfc7bada909dc5cc228211b075ba8407c46467f63597c78178",
                "sha256:1b9362d34363f2c71b7853f6251219298124aa4cc2075ae2932e64c91a3e2717",
                "sha256:1f3bfbd611db5cb58ca82f3deb35e83af34bb8cf06043fa61500157d50a70982",
                "sha256:2bd1096476aaac820426239ab534b636c77d71af66c547b9ddcd76eb9c79e004",
                "sha256:31fe38d14d2e5f787e0aecef831457da6cec68e0bb09a35835b0b44ae8b988fe",
                "sha256:3b8398b3d0efc420e777c40c16764d6870bcef2eb383df9c6dbb9ffe12c64452",
                "sha256:3c81599befb4d4f3d7648ed3217e00d21a9341a9a688ecdd615ff72ffbed7336",
                "sha256:419c57d7b63f5ec38b1199a9521d77d7d1754eb97827bbb773162073ccd8c8d4",
                "sha256:46f4c544f6557a2fefa7ac8ac7d1b17bf9b647bd20b16decc8fbcab7117fbc15",
                "sha256:471e0d70201c069f74c837983189949aa0d24bb2d751b57e26e3761f2f782b8d",
                "sha256:59b281eab51e1b6b6afa525af2bd93c16d49358404f814fe2c2410058623928c",
                "sha256:731c8abd27693323b348518ed0e0705713a36d79fdbd969ad968fbef0979a7e0",
                "sha256:95e590dd70642eb2079d280420a888190aa040ad20f19ec8c6e097e38aa29e06",
                "sha256:a68254dd88021f24a68b613d8c51d5c5e74d735878b9e32cc0adf19d1f10aaf9",
                "sha256:a7d5137e556cc0ea418dca6186deabe9129cee318618eb1ffecbd35bee55ddc1",
                "sha256:aeaba7b5e756ea52c8861c133c596afe93dd716cbcacae23b80bc238202dc023",
                "sha256:dc26bb134452081859aa21d4990474ddb7e863aa39e60d1592800a8865a702de",
                "sha256:e53258e69874a306fcecb88b7534d61820db8a98655662a3dd2ec7f1afd9132f",
                "sha256:ef15c2df7656763b4ff20a9bc4381d8352e6640cfeb95c2972c38ef508e75181",
                "sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e",
                "sha256:f8ec91983e638a9bcd75b39f1396e5c0dc2330cbd9ce4accefe68717e6779e0a"
            ],
            "index": "pypi",
            "version": "==37.0.2"
        },

dijital20 commented 1 year ago

For cryptography, there's a compiled component written in Rust which is packaged along with the Python code. This would be different per-platform, which accounts for the different hashes per platform. This page on PyPI shows the different platform variations of the files:

https://pypi.org/project/cryptography/#files

The lock you posted @matteius looks like it got all of them. What @xyxz-web is seeing are probably the ones appropriate for the platform and interpreter they're using.

Like I said, I have a colleague who is trying to author a new package and wants to support both Windows and Linux, so he's developing out of both Windows and WSL2 Ubuntu. At least one of the dependencies (I suspect lxml) is different between the platforms, so if he locks on one platform, the lock file doesn't work on the other and vice versa.

We have changed to using lock on Windows, and then lock --keep-outdated on the Linux, but there's got to be an easier, slightly more obvious way.

I hope this is helpful. :) Thank you for the discussion on this.

matteius commented 1 year ago

@dijital20 You are right that locked that on windows, but I just tried on Ubuntu and got the same result:

        "cryptography": {
            "hashes": [
                "sha256:093cb351031656d3ee2f4fa1be579a8c69c754cf874206be1d4cf3b542042804",
                "sha256:0cc20f655157d4cfc7bada909dc5cc228211b075ba8407c46467f63597c78178",
                "sha256:1b9362d34363f2c71b7853f6251219298124aa4cc2075ae2932e64c91a3e2717",
                "sha256:1f3bfbd611db5cb58ca82f3deb35e83af34bb8cf06043fa61500157d50a70982",
                "sha256:2bd1096476aaac820426239ab534b636c77d71af66c547b9ddcd76eb9c79e004",
                "sha256:31fe38d14d2e5f787e0aecef831457da6cec68e0bb09a35835b0b44ae8b988fe",
                "sha256:3b8398b3d0efc420e777c40c16764d6870bcef2eb383df9c6dbb9ffe12c64452",
                "sha256:3c81599befb4d4f3d7648ed3217e00d21a9341a9a688ecdd615ff72ffbed7336",
                "sha256:419c57d7b63f5ec38b1199a9521d77d7d1754eb97827bbb773162073ccd8c8d4",
                "sha256:46f4c544f6557a2fefa7ac8ac7d1b17bf9b647bd20b16decc8fbcab7117fbc15",
                "sha256:471e0d70201c069f74c837983189949aa0d24bb2d751b57e26e3761f2f782b8d",
                "sha256:59b281eab51e1b6b6afa525af2bd93c16d49358404f814fe2c2410058623928c",
                "sha256:731c8abd27693323b348518ed0e0705713a36d79fdbd969ad968fbef0979a7e0",
                "sha256:95e590dd70642eb2079d280420a888190aa040ad20f19ec8c6e097e38aa29e06",
                "sha256:a68254dd88021f24a68b613d8c51d5c5e74d735878b9e32cc0adf19d1f10aaf9",
                "sha256:a7d5137e556cc0ea418dca6186deabe9129cee318618eb1ffecbd35bee55ddc1",
                "sha256:aeaba7b5e756ea52c8861c133c596afe93dd716cbcacae23b80bc238202dc023",
                "sha256:dc26bb134452081859aa21d4990474ddb7e863aa39e60d1592800a8865a702de",
                "sha256:e53258e69874a306fcecb88b7534d61820db8a98655662a3dd2ec7f1afd9132f",
                "sha256:ef15c2df7656763b4ff20a9bc4381d8352e6640cfeb95c2972c38ef508e75181",
                "sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e",
                "sha256:f8ec91983e638a9bcd75b39f1396e5c0dc2330cbd9ce4accefe68717e6779e0a"
            ],
            "index": "pypi",
            "version": "==37.0.2"
        },

I am going to need y'all to provide the output from pipenv --support to further debug,

ghost commented 1 year ago

I also get the long list of hashes when I lock from within WSL - seems to include both Windows and Linux hashes. However, when I lock from within Windows 10 or a Linux VM, I get a much shorter list. I'm usinge a private repo here, but it is automatically mirrored from pypi.

matteius commented 1 year ago

@xyxz-web hash collection of all packages is a specific feature of pypi. There is no API into private pypi servers for collecting the set of hashes, and I suspect that is where the problem comes in. For reference, here is the code that collects the hashes: https://github.com/pypa/pipenv/blob/main/pipenv/utils/resolver.py#L757-L788 Specifically look for where it collects hashes from pypi: https://github.com/pypa/pipenv/blob/main/pipenv/utils/resolver.py#L726-L755

I would recommend if possible to use pypi as a primary index in the Pipfile and support your private packages as a secondary index calling out specifically the private index to pull those specific packages from.

januszm commented 1 year ago

how to know which one to use during sync

@matteius perhaps with:

import platform

# somwehere inside the `sync` method definition
open(f"Pipfile.lock.{platform.machine()}")

I would not include the Python version in the lockfile name, project should be developed for the version running on the production servers to avoid surprises and provide more or less stable development experience and features provided by the specific version of the language used.

januszm commented 1 year ago

I think this is a really important problem to solve right now. For example, if we have a project that uses numpy as a dependency and we have an x86_64 server, Github Actions CI also on x86_64, 1 developer on Linux x86_64, one on older x86_64 Mac and one more with an M1 Macbook arm64. Each time someone updates the lockfile, at least one person has problem. At the moment, one quick solution I found was to manually modify the Pipfile.lock file and add this (for numpy):

"markers": "(platform_machine == 'x86_64' or platform_machine == 'arm64') and python_version < '3.10'",

in

"numpy": {
  "hashes": [...],
  "markers": "...",
  "version": "==1.23.2"

(the above allows me to work on M1 Mac and run CI on x86_64 Github Action + deploy to amd64 Linux server, but it's not sustainable as next time someone adds or updates a package it'll be lost)

The Ruby Bundler solves this problem by adding separate lock file entries for each platform, e.g.

nokogiri (1.13.6)
nokogiri (1.13.6-aarch64-linux)
nokogiri (1.13.6-x86_64-darwin)
nokogiri (1.13.6-x86_64-linux)

matteius commented 1 year ago

I think this is a really important problem to solve right now.

I agree with you, it is becoming higher on the list of priorities and we have tackled a number of other bugs and technical debt items that set us up better to work on this. I am inclined to try and find a solution that will work with the single Pipfile.lock than rather try to support multiple lock files by platform. I am not quite sure how it would work though for something that has to be built on that system architecture in order to generate the appropriate hashes because how could it build all of the architectures at once? That is probably where the thinking to have a separate platform specific lockfiles comes into play but I need to give this problem more time and consideration as well as additional inputs will be helpful.

@januszm Can you explain more about your workaround "markers": "(platform_machine == 'x86_64' or platform_machine == 'arm64') and python_version < '3.10'", -- can you provide more context, maybe the whole block from that part of the lockfile so I can get a better idea of what the workaround is doing.

Another thing I have been thinking more about as it relates to pre-built wheels and private pypi's, is it would be great if we could get the private pypi servers to support a similar json API to pypi for fetching the available package hashes. Of the data that API returns, pipenv specifically leverages the part of the payload that is: api_releases = r.json()["releases"] to have the versions as keys with each having a key ["digests"]["sha256"] which would be the package hash for that version package. If we knew that more private pypi servers supported this informational package API, then we could update our logic here to try this for more than just pypi.org: https://github.com/pypa/pipenv/blob/main/pipenv/utils/resolver.py#L727-L755

That wouldn't help for packages that need to be built on system architectures to get those hashes (because no pre-built wheels exist) but it would help in general with this issue and with performance, though a platform specific build solution I believe needs to exist as well to manage hashes generated from different lock architectures. I think the numpy issue though, there are pre-built wheels for most architectures and pipenv will not pull all of the hashes from a private pypi because it would have to download all of the wheels because such an API is not supported. Perhaps part of the solution could be an option to override and tell it to download all platform eligible wheels from a private pypi in order to collect all the hashes, but again I would need to study that code some more to find out.

januszm commented 1 year ago

Thanks for the detailed response @matteius . I don't know if I will be able to professionally explain what exactly this workaround is because I have never studied pip and pipenv internals, I am rather just a user of these systems. As I understand it, this line, adding or platform_machine == 'arm64' just lets pip build a local extension for numpy, it unlocks the installation process on arm64 platform.

I am not sure how hashes work in pip and pipenv but IMO it seems unnecessary to calculate hashes for extensions that are built locally. The hashes for the source code, which is common to all architectures, should provide necessary minimum security so it can be computed once. Probably that's why it works with my workaround, the hash of files that are built locally doesn't matter because they come from source code that already has hashes.

UPDATE: I just received this from a friend who uses poetry to manage dependencies in the project. Looks like poetry solves this issue by storing a list of specifications for each platform for a particular dependency:

[package.dependencies]
numpy = [
    {version = ">=1.18.5", markers = "platform_machine != \"aarch64\" and platform_machine != \"arm64\" and python_version < \"3.10\""},
    {version = ">=1.19.2", markers = "platform_machine == \"aarch64\" and python_version < \"3.10\""},
    {version = ">=1.20.0", markers = "platform_machine == \"arm64\" and python_version < \"3.10\""},
    {version = ">=1.21.0", markers = "python_version >= \"3.10\""},
]
python-dateutil = ">=2.8.1"
# ...

matteius commented 1 year ago

@januszm Thanks I will take a closer look at that later, fwiw, I think you may be actually installing pre-built wheels for the platform. For example, there are prebuilt arm64 wheels for macos: https://pypi.org/project/numpy/1.23.2/#files

The issue for pipenv though for pre-built wheels is when going through a private pypi, I believe it does not download all of the wheels to get the hashes, only the ones that match the system you are locking on.

januszm commented 1 year ago

I think you may be actually installing pre-built wheels for the platform

Ah true, if we could make pipenv work with all platforms, not only the one we're locking on, it would already be a great improvement

drorata commented 1 year ago

TL;DR another case where this issue has a negative impact is when trying to publish a Jupyter notebook using mybinder.

Details: I have a notebook that I develop locally (M1 mac) and I'm trying to serve it using https://mybinder.org/. During the build it seems like numpy was not installed due to the platform specified in the lock file. In turn, the served notebook fails... See this discussion for more details.

Possible workaround: In https://github.com/drorata/candy-analysis/commit/ed2c6eb46f1fb740a0efee3d208ebe92dcc5aeef I removed the Pipfile.lock so when the image will be built on mybinder's side, it will be free to pick the appropriate platform. This unblocked my build and running of the notebook inside mybinder.

drorata commented 1 year ago

One addition, FWIW: I now faced a similar related issue. I'm trying to dockerize some work which was developed on an M1 mac. To that end, in the Dockerfile I added:

COPY ./Pipfile* .

RUN python -m pip install --upgrade pip
RUN pip install pipenv && pipenv install --dev --system --deploy

It comes as no surprise, that I get the following warning: Ignoring appnope: markers 'platform_system == "Darwin"' don't match your environment. I tried not to copy the Pipfile.lock, but then it is not possible to use the --system flag when installing the environment. I don't think it harmed the running of the container but I find it very confusing.

matteius commented 1 year ago

@drorata I am not sure how the markers are making it into your Pipfile for numpy ... I just locked on many different systems numpy, on windows, linux VM, Mac M1, and all did not have the markers restricting numpy. What version of pipenv are you on?

matteius commented 1 year ago

Also noting to the larger group that I am unable to reproduce getting not the full set of hashes for some of these packages on pipenv==2022.8.24 -- I tried multiple systems and I also tried pypiserver which mirrors pypi packages, and in all cases the lock file has the full set of hashes. It appears even when it has to download the wheels, it downloads all of the matching version to get the hashes. If anyone can provide more reproduction steps with the current version of pipenv I would like to look more into it.

drorata commented 1 year ago

@matteius Thanks for your answer! I'm used version 2022.8.17 (installed using brew). I upgraded it (to 2022.8.24) and now there are no markers in the Pipfile.lock. In turn the warning doesn't show up.

januszm commented 1 year ago

Does it mean this issue was fixed in 24.08?

EDIT: looks like it does, after the pipenv upgrade I see that the platform specific markers are gone. Example for numpy:

"markers": "python_version < '3.10'",
"version": "==1.23.2"

drorata commented 1 year ago

Does it mean this issue was fixed in 24.08?

That's a bold statement @januszm. I'm not sure what's meant by "this issue". In my case, using the newer version yielded a Pipfile.lock file which didn't contain the "markers" and I was able to build the docker image without warnings.

thehesiod commented 1 year ago

I think it really needs to be one lockfile per platform because of stuff like: https://github.com/pandas-dev/pandas/blob/v1.4.3/setup.cfg#L33, which each platform has a different version requirement. Otherwise locks would have to munge a single lockfile which could prove problematic due to competing requirements

thehesiod commented 1 year ago

I think it's worth looking at the poetry file format for this type of scenario:

[package.dependencies]
numpy = [
    {version = ">=1.18.5", markers = "platform_machine != \"aarch64\" and platform_machine != \"arm64\" and python_version < \"3.10\""},
    {version = ">=1.19.2", markers = "platform_machine == \"aarch64\" and python_version < \"3.10\""},
    {version = ">=1.20.0", markers = "platform_machine == \"arm64\" and python_version < \"3.10\""},
]

along with

numpy = [
    {file = "numpy-1.23.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:aeba539285dcf0a1ba755945865ec61240ede5432df41d6e29fab305f4384db2"},
    {file = "numpy-1.23.1-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:7e8229f3687cdadba2c4faef39204feb51ef7c1a9b669247d49a24f3e2e1617c"},
    {file = "numpy-1.23.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1408c3527a74a0209c781ac82bde2182b0f0bf54dea6e6a363fe0cc4488a7ce7"},
    {file = "numpy-1.23.1.tar.gz", hash = "sha256:d748ef349bfef2e1194b59da37ed5a29c19ea8d7e6342019921ba2ba4fd8b624"},
]

that way multiple versions are supported, with each platform getting what it requires

matteius commented 1 year ago

@thehesiod That is really complicated, I mean perhaps possible to specify all that to the pip resolver, but never the less couldn't your example be boiled down to this for pipenv locking and work properly?

numpy = {version = ">=1.20.0", markers = "python_version < '3.10'"}

Since if you specify all three constraints for version to the resolver, you are going to have to get one version that is compatible across all constraints, so you may as well go with version = ">=1.20.0" and then I guess the question becomes what are you trying to accomplish with all those marker constarints? It is not clear from the example, but the above line resolves a lock file entry like:

        "numpy": {
            "hashes": [
                "sha256:17e5226674f6ea79e14e3b91bfbc153fdf3ac13f5cc54ee7bc8fdbe820a32da0",
                "sha256:2bd879d3ca4b6f39b7770829f73278b7c5e248c91d538aab1e506c628353e47f",
                "sha256:4f41f5bf20d9a521f8cab3a34557cd77b6f205ab2116651f12959714494268b0",
                "sha256:5593f67e66dea4e237f5af998d31a43e447786b2154ba1ad833676c788f37cde",
                "sha256:5e28cd64624dc2354a349152599e55308eb6ca95a13ce6a7d5679ebff2962913",
                "sha256:633679a472934b1c20a12ed0c9a6c9eb167fbb4cb89031939bfd03dd9dbc62b8",
                "sha256:806970e69106556d1dd200e26647e9bee5e2b3f1814f9da104a943e8d548ca38",
                "sha256:806cc25d5c43e240db709875e947076b2826f47c2c340a5a2f36da5bb10c58d6",
                "sha256:8247f01c4721479e482cc2f9f7d973f3f47810cbc8c65e38fd1bbd3141cc9842",
                "sha256:8ebf7e194b89bc66b78475bd3624d92980fca4e5bb86dda08d677d786fefc414",
                "sha256:8ecb818231afe5f0f568c81f12ce50f2b828ff2b27487520d85eb44c71313b9e",
                "sha256:8f9d84a24889ebb4c641a9b99e54adb8cab50972f0166a3abc14c3b93163f074",
                "sha256:909c56c4d4341ec8315291a105169d8aae732cfb4c250fbc375a1efb7a844f8f",
                "sha256:9b83d48e464f393d46e8dd8171687394d39bc5abfe2978896b77dc2604e8635d",
                "sha256:ac987b35df8c2a2eab495ee206658117e9ce867acf3ccb376a19e83070e69418",
                "sha256:b78d00e48261fbbd04aa0d7427cf78d18401ee0abd89c7559bbf422e5b1c7d01",
                "sha256:b8b97a8a87cadcd3f94659b4ef6ec056261fa1e1c3317f4193ac231d4df70215",
                "sha256:bd5b7ccae24e3d8501ee5563e82febc1771e73bd268eef82a1e8d2b4d556ae66",
                "sha256:bdc02c0235b261925102b1bd586579b7158e9d0d07ecb61148a1799214a4afd5",
                "sha256:be6b350dfbc7f708d9d853663772a9310783ea58f6035eec649fb9c4371b5389",
                "sha256:c403c81bb8ffb1c993d0165a11493fd4bf1353d258f6997b3ee288b0a48fce77",
                "sha256:cf8c6aed12a935abf2e290860af8e77b26a042eb7f2582ff83dc7ed5f963340c",
                "sha256:d98addfd3c8728ee8b2c49126f3c44c703e2b005d4a95998e2167af176a9e722",
                "sha256:dc76bca1ca98f4b122114435f83f1fcf3c0fe48e4e6f660e07996abf2f53903c",
                "sha256:dec198619b7dbd6db58603cd256e092bcadef22a796f778bf87f8592b468441d",
                "sha256:df28dda02c9328e122661f399f7655cdcbcf22ea42daa3650a26bce08a187450",
                "sha256:e603ca1fb47b913942f3e660a15e55a9ebca906857edfea476ae5f0fe9b457d5",
                "sha256:ecfdd68d334a6b97472ed032b5b37a30d8217c097acfff15e8452c710e775524"
            ],
            "markers": "python_version < '3.10'",
            "version": "==1.23.2"
        },

This includes all of the relevant hashes for this version. Is there a platform you are trying to exclude numpy from?

matteius commented 1 year ago

I spent some time on this today and what I found was somewhat discouraging -- internal to the pip resolver the markers of transitive dependencies are processed and those requirements are discarded if they are not for your platform. Pipenv has such a thing in its lock file: it depends on click which has metadata:

Requires-Dist: colorama ; platform_system == "Windows"
Requires-Dist: importlib-metadata ; python_version < "3.8"

So if we do not lock on Windows, the colorama is dropped from the lock file. The problem with my code modifications had nothing to do with colorama, but other dependencies that did not have wheell/metadata for my system and so it tries to build that metadata but fails to do so because the marker was ignored that indicates it was not for my system. I tried to hack around it, but I ended up in long locking phases that never ended, or ones that ended because some dependencie's possible extra could not generate the metadata on my system.

The interesting thing though, is by adding a resolution of that same package to the Pipfile, then the lock does work on linux and does not drop the colorama. For example adding: `colorama = {version = "", markers="sys_platform == 'win32'"}to thePipfileof pipenv and locking on linux does not change thecoloarama` in the lock file. I note this as a workaround to a fairly complex problem that I don't see an easy solution for. It is interesting though, requirementslib has a setup.py reader that generates some kind of metadata without invoking any of the setup.py logic, which seems quite different from what the pip resolver is doing--the pip resolver I believe is designed to prep packages for install and so perhaps it is reusing logic for getting the metadata that assumes the package will definitely be installed on that system, when what we want is to be able to resolve a proper lockfile that includes all platforms with the relevant markers.

matteius commented 1 year ago

I am thinking that named categories may help out with this issue -- See: https://github.com/pypa/pipenv/pull/5366

For example:

$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "*"

[dev-packages]

[requires]
python_version = "3.10"

[linux]
numpy = "*"

[windows]
numpy = "==1.22.4"

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/multi-platform
$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "18815ba6d0514cb0588df0a7bc63bca22ea99223fc0ba15281b0258dbefff304"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {},
    "linux": {
        "numpy": {
            "hashes": [
                "sha256:004f0efcb2fe1c0bd6ae1fcfc69cc8b6bf2407e0f18be308612007a0762b4089",
                "sha256:09f6b7bdffe57fc61d869a22f506049825d707b288039d30f26a0d0d8ea05164",
                "sha256:0ea3f98a0ffce3f8f57675eb9119f3f4edb81888b6874bc1953f91e0b1d4f440",
                "sha256:17c0e467ade9bda685d5ac7f5fa729d8d3e76b23195471adae2d6a6941bd2c18",
                "sha256:1f27b5322ac4067e67c8f9378b41c746d8feac8bdd0e0ffede5324667b8a075c",
                "sha256:22d43376ee0acd547f3149b9ec12eec2f0ca4a6ab2f61753c5b29bb3e795ac4d",
                "sha256:2ad3ec9a748a8943e6eb4358201f7e1c12ede35f510b1a2221b70af4bb64295c",
                "sha256:301c00cf5e60e08e04d842fc47df641d4a181e651c7135c50dc2762ffe293dbd",
                "sha256:39a664e3d26ea854211867d20ebcc8023257c1800ae89773cbba9f9e97bae036",
                "sha256:51bf49c0cd1d52be0a240aa66f3458afc4b95d8993d2d04f0d91fa60c10af6cd",
                "sha256:78a63d2df1d947bd9d1b11d35564c2f9e4b57898aae4626638056ec1a231c40c",
                "sha256:7cd1328e5bdf0dee621912f5833648e2daca72e3839ec1d6695e91089625f0b4",
                "sha256:8355fc10fd33a5a70981a5b8a0de51d10af3688d7a9e4a34fcc8fa0d7467bb7f",
                "sha256:8c79d7cf86d049d0c5089231a5bcd31edb03555bd93d81a16870aa98c6cfb79d",
                "sha256:91b8d6768a75247026e951dce3b2aac79dc7e78622fc148329135ba189813584",
                "sha256:94c15ca4e52671a59219146ff584488907b1f9b3fc232622b47e2cf832e94fb8",
                "sha256:98dcbc02e39b1658dc4b4508442a560fe3ca5ca0d989f0df062534e5ca3a5c1a",
                "sha256:a64403f634e5ffdcd85e0b12c08f04b3080d3e840aef118721021f9b48fc1460",
                "sha256:bc6e8da415f359b578b00bcfb1d08411c96e9a97f9e6c7adada554a0812a6cc6",
                "sha256:bdc9febce3e68b697d931941b263c59e0c74e8f18861f4064c1f712562903411",
                "sha256:c1ba66c48b19cc9c2975c0d354f24058888cdc674bebadceb3cdc9ec403fb5d1",
                "sha256:c9f707b5bb73bf277d812ded9896f9512a43edff72712f31667d0a8c2f8e71ee",
                "sha256:d5422d6a1ea9b15577a9432e26608c73a78faf0b9039437b075cf322c92e98e7",
                "sha256:e5d5420053bbb3dd64c30e58f9363d7a9c27444c3648e61460c1237f9ec3fa14",
                "sha256:e868b0389c5ccfc092031a861d4e158ea164d8b7fdbb10e3b5689b4fc6498df6",
                "sha256:efd9d3abe5774404becdb0748178b48a218f1d8c44e0375475732211ea47c67e",
                "sha256:f8c02ec3c4c4fcb718fdf89a6c6f709b14949408e8cf2a2be5bfa9c49548fd85",
                "sha256:ffcf105ecdd9396e05a8e58e81faaaf34d3f9875f137c7372450baa5d77c9a54"
            ],
            "index": "pypi",
            "version": "==1.23.3"
        }
    },
    "windows": {
        "numpy": {
            "hashes": [
                "sha256:0791fbd1e43bf74b3502133207e378901272f3c156c4df4954cad833b1380207",
                "sha256:1ce7ab2053e36c0a71e7a13a7475bd3b1f54750b4b433adc96313e127b870887",
                "sha256:2d487e06ecbf1dc2f18e7efce82ded4f705f4bd0cd02677ffccfb39e5c284c7e",
                "sha256:37431a77ceb9307c28382c9773da9f306435135fae6b80b62a11c53cfedd8802",
                "sha256:3e1ffa4748168e1cc8d3cde93f006fe92b5421396221a02f2274aab6ac83b077",
                "sha256:425b390e4619f58d8526b3dcf656dde069133ae5c240229821f01b5f44ea07af",
                "sha256:43a8ca7391b626b4c4fe20aefe79fec683279e31e7c79716863b4b25021e0e74",
                "sha256:4c6036521f11a731ce0648f10c18ae66d7143865f19f7299943c985cdc95afb5",
                "sha256:59d55e634968b8f77d3fd674a3cf0b96e85147cd6556ec64ade018f27e9479e1",
                "sha256:64f56fc53a2d18b1924abd15745e30d82a5782b2cab3429aceecc6875bd5add0",
                "sha256:7228ad13744f63575b3a972d7ee4fd61815b2879998e70930d4ccf9ec721dce0",
                "sha256:9ce7df0abeabe7fbd8ccbf343dc0db72f68549856b863ae3dd580255d009648e",
                "sha256:a911e317e8c826ea632205e63ed8507e0dc877dcdc49744584dfc363df9ca08c",
                "sha256:b89bf9b94b3d624e7bb480344e91f68c1c6c75f026ed6755955117de00917a7c",
                "sha256:ba9ead61dfb5d971d77b6c131a9dbee62294a932bf6a356e48c75ae684e635b3",
                "sha256:c1d937820db6e43bec43e8d016b9b3165dcb42892ea9f106c70fb13d430ffe72",
                "sha256:cc7f00008eb7d3f2489fca6f334ec19ca63e31371be28fd5dad955b16ec285bd",
                "sha256:d4c5d5eb2ec8da0b4f50c9a843393971f31f1d60be87e0fb0917a49133d257d6",
                "sha256:e96d7f3096a36c8754207ab89d4b3282ba7b49ea140e4973591852c77d09eb76",
                "sha256:f0725df166cf4785c0bc4cbfb320203182b1ecd30fee6e541c8752a92df6aa32",
                "sha256:f3eb268dbd5cfaffd9448113539e44e2dd1c5ca9ce25576f7c04a5453edc26fa",
                "sha256:fb7a980c81dd932381f8228a426df8aeb70d59bbcda2af075b627bbc50207cba"
            ],
            "index": "pypi",
            "version": "==1.22.4"
        }
    }
}

Then you can install those specific platform lock sections: pipenv sync --categories windows

thehesiod commented 1 year ago

@matteius for that example yes, it just points out that each platform could have a totally unique dependency, like mac = 1.0, windows = 2.0, linux=3.0. You can't always simplify :) So the poetry way of maintaining all the requirements and re-processing them during sync is more flexible

matteius commented 1 year ago

@thehesiod The problem is you cannot create the pipenv locks on the system you are not on for transitive dependencies or sdists that need to be built on that specific platform, so maintaining all of the requirements in one section is less practical to how pipenv utilizes the pip resolver.

thehesiod commented 1 year ago

@matteius ya, it would be nice to have the lockfiles be per arch, perhaps like Pipfile.w32.lock or something. Given you can have runtime dynamic dependencies it makes more sense to have one lockfile per arch, perhaps like Pipfile.darwin_x86_64.lock or something. Actually I just found a counterpoint with poetry, it's unable to lock if not all platforms have valid scenarios, like we don't do win32, but because of this line it fails: https://github.com/PyCQA/pylint/blob/v2.15.2/pyproject.toml#L45.

drmikecrowe commented 1 year ago

I've tried to find a way to detect if this could be an issue in my Pipfile.lock. I think the following command detects this. Wouldn't it?

pipenv run grep "Root-Is-Purelib: false" $(dirname $(dirname $(which python3)))/lib/python3*/site-packages * -R --include=WHEEL

matteius commented 1 year ago

Just wanted to update this thread that named package categories were released in October and some users have found it a good way to manage multi-platform dependencies for a single project (such as developers on different operating systems all using pytorch) -- https://pipenv.pypa.io/en/latest/basics/#specifying-package-categories

I would suggest trying to make use of package categories to solve these edge cases, as there are not current plans to support lock file per platform.

januszm commented 1 year ago

@matteius thanks for the update, however I think this section of the doc should contain more details about how this should actually be used to support multiple platforms

This may be desired in some use cases where users only are installing groups specific to their system platform

Eg. How to define groups for arm64 and x86_64 and how to use them to install packages

matteius commented 1 year ago

@januszm That is a good suggestion to improve the docs around this -- I think in general the docs stand to be improved. You may find this example helpful: https://github.com/pypa/pipenv/issues/5455#issuecomment-1304063926

I'd also like inputs about other options or configuration variables that would help make using the named categories easier -- for example right now pipenv lock --categories="default dev arm64" would lock only those packages, but maybe an option topipenv lock --exclude-categories=x86_64would be useful, and a config var that could be put into the.envthat always applies a categories or exclude-categories string to make working with thePipfile.lock` easier when working cross-platform.

januszm commented 1 year ago

Thanks, I'm not sure if I understand correctly how this will work, but does it mean that people working on different architectures will generate different sections of the Pipfile.lock but will not interrupt each others groups ? So if developer A adds numpy on x86_64, she changes Pipfile.lock, then developer B on Macbook M1 also adds numpy, arm64 group is added and Pipfile.lock contains both variants? (from now on all developers can pipenv sync and get numpy for their platform).

I just hope that this way of installing an architecture-dependent package via lock won't get all other packages updated (a memorable --keep-outdated issue) - the point is to be able to make "atomic" changes to lockfile, important for effective debugging and version control.

kolibri91 commented 1 year ago

@matteius Thanks for sharing your release including the named categories all over the place. It's a nice feature but it does not solve the "multi-platform" problem. It's now possible to define packages per category but that's already all. I'm still facing quite similar problems like @januszm.

I have a setup with some tools in the dev-section of the Pipfile. It works great till those tools does not have platform-dependent dependencies. You might think that's quite rare but think about colorama. Almost every code-quality-tool like ( black, pytest, pylint, etc.) needs this package to print colors on windows system. This package is not required on linux systems. What will now happen if one user locks on windows and the other on linux? Exactly, every lock creates a new lock-file, one with and one without colorama. It's possible to solve this issue by adding colorama to the Pipfile with a platform-marker (like @matteius suggested) but there are other examples like keyring of twine where it not works because it includes several dependencies over multiple levels. It will be a mess to maintain this by hand.

To keep all those packages in the lock-file, the --keep-outdated flag must be set but then it's almost impossible to keep the packages up-to-date or just upgrade one certain package.

The only way to go I currently see would be a complete duplication of the whole "dev"-section per platform. Is this really, what you propose with the "named categories"?

matteius commented 1 year ago

Can I get some more examples of cross platform locking problems that don't involve wanting different versions of libraries (which technically can be handled by categories) -- I've been playing around with our resolver implementation and I see a way to specify the platforms to the finder we create in a new method for the finder I've worked on but I don't have great examples to test with.

@kolibri91 Can you explain more about the abut the twine or keyring examples?

kolibri91 commented 1 year ago

Hi, of course. Please take a look at the following minimal example. I created a minimalistic Pipfile and executed pipenv lock --dev on windows and linux platform with latest version pipenv, version 2023.3.20. I attached the generated lock-files to this post, just remove the unnecessary suffixes.

[requires]
python_version = "3.8"

[dev-packages]
twine = "*"

[packages]
numpy = "*"

Pipfile.lock.win64.log Pipfile.lock.linux.log

nk9 commented 1 year ago

I believe I'm running into this as well: https://github.com/pypa/pipenv/issues/5723

I have a package which only works on Linux, but development is happening on an M1 Mac. Because pipenv is skipping the package entirely, there are no hashes in the lockfile. When I attempt to install --deploy using the lockfile on Linux, the install fails.

fredizzimo commented 11 months ago

For colorama it looks like the issue is that it uses platform_system, while many other packages use sys_platform and they work on other platforms.

Why do those work differently?

matteius commented 10 months ago

X-posting this for visibility, a recent issue lead to me working on a path forward for cross platform dependencies, and I think I have something that could stick. https://github.com/pypa/pipenv/pull/5892

Its probably only consider sys_platform markers right now, but I think it makes sense to apply it to all marker types.

Please have a look ... here is an example:

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-bug-demo (main)
$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = {version = "*", sys_platform = "== 'linux'"}
urllib3 = "*"
#requests = {version = "*", markers="sys_platform == 'linux'"}

[dev-packages]

[requires]
python_version = "3.11"

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-bug-demo (main)
$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "132a5eb6d6138353e2837c95af6700c90651a30015f73b576a8a0268573d3548"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.11"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:539cc1d13202e33ca466e88b2807e29f4c13049d6d87031a3c110744495cb082",
                "sha256:92d6037539857d8206b8f6ae472e8b77db8058fec5937a1ef3f54304089edbb9"
            ],
            "markers": "python_version >= '3.6' and sys_platform == 'linux'",
            "version": "==2023.7.22"
        },
        "charset-normalizer": {
            "hashes": [
                "sha256:04e57ab9fbf9607b77f7d057974694b4f6b142da9ed4a199859d9d4d5c63fe96",
                "sha256:09393e1b2a9461950b1c9a45d5fd251dc7c6f228acab64da1c9c0165d9c7765c",
                "sha256:0b87549028f680ca955556e3bd57013ab47474c3124dc069faa0b6545b6c9710",
                "sha256:1000fba1057b92a65daec275aec30586c3de2401ccdcd41f8a5c1e2c87078706",
                "sha256:1249cbbf3d3b04902ff081ffbb33ce3377fa6e4c7356f759f3cd076cc138d020",
                "sha256:1920d4ff15ce893210c1f0c0e9d19bfbecb7983c76b33f046c13a8ffbd570252",
                "sha256:193cbc708ea3aca45e7221ae58f0fd63f933753a9bfb498a3b474878f12caaad",
                "sha256:1a100c6d595a7f316f1b6f01d20815d916e75ff98c27a01ae817439ea7726329",
                "sha256:1f30b48dd7fa1474554b0b0f3fdfdd4c13b5c737a3c6284d3cdc424ec0ffff3a",
                "sha256:203f0c8871d5a7987be20c72442488a0b8cfd0f43b7973771640fc593f56321f",
                "sha256:246de67b99b6851627d945db38147d1b209a899311b1305dd84916f2b88526c6",
                "sha256:2dee8e57f052ef5353cf608e0b4c871aee320dd1b87d351c28764fc0ca55f9f4",
                "sha256:2efb1bd13885392adfda4614c33d3b68dee4921fd0ac1d3988f8cbb7d589e72a",
                "sha256:2f4ac36d8e2b4cc1aa71df3dd84ff8efbe3bfb97ac41242fbcfc053c67434f46",
                "sha256:3170c9399da12c9dc66366e9d14da8bf7147e1e9d9ea566067bbce7bb74bd9c2",
                "sha256:3b1613dd5aee995ec6d4c69f00378bbd07614702a315a2cf6c1d21461fe17c23",
                "sha256:3bb3d25a8e6c0aedd251753a79ae98a093c7e7b471faa3aa9a93a81431987ace",
                "sha256:3bb7fda7260735efe66d5107fb7e6af6a7c04c7fce9b2514e04b7a74b06bf5dd",
                "sha256:41b25eaa7d15909cf3ac4c96088c1f266a9a93ec44f87f1d13d4a0e86c81b982",
                "sha256:45de3f87179c1823e6d9e32156fb14c1927fcc9aba21433f088fdfb555b77c10",
                "sha256:46fb8c61d794b78ec7134a715a3e564aafc8f6b5e338417cb19fe9f57a5a9bf2",
                "sha256:48021783bdf96e3d6de03a6e39a1171ed5bd7e8bb93fc84cc649d11490f87cea",
                "sha256:4957669ef390f0e6719db3613ab3a7631e68424604a7b448f079bee145da6e09",
                "sha256:5e86d77b090dbddbe78867a0275cb4df08ea195e660f1f7f13435a4649e954e5",
                "sha256:6339d047dab2780cc6220f46306628e04d9750f02f983ddb37439ca47ced7149",
                "sha256:681eb3d7e02e3c3655d1b16059fbfb605ac464c834a0c629048a30fad2b27489",
                "sha256:6c409c0deba34f147f77efaa67b8e4bb83d2f11c8806405f76397ae5b8c0d1c9",
                "sha256:7095f6fbfaa55defb6b733cfeb14efaae7a29f0b59d8cf213be4e7ca0b857b80",
                "sha256:70c610f6cbe4b9fce272c407dd9d07e33e6bf7b4aa1b7ffb6f6ded8e634e3592",
                "sha256:72814c01533f51d68702802d74f77ea026b5ec52793c791e2da806a3844a46c3",
                "sha256:7a4826ad2bd6b07ca615c74ab91f32f6c96d08f6fcc3902ceeedaec8cdc3bcd6",
                "sha256:7c70087bfee18a42b4040bb9ec1ca15a08242cf5867c58726530bdf3945672ed",
                "sha256:855eafa5d5a2034b4621c74925d89c5efef61418570e5ef9b37717d9c796419c",
                "sha256:8700f06d0ce6f128de3ccdbc1acaea1ee264d2caa9ca05daaf492fde7c2a7200",
                "sha256:89f1b185a01fe560bc8ae5f619e924407efca2191b56ce749ec84982fc59a32a",
                "sha256:8b2c760cfc7042b27ebdb4a43a4453bd829a5742503599144d54a032c5dc7e9e",
                "sha256:8c2f5e83493748286002f9369f3e6607c565a6a90425a3a1fef5ae32a36d749d",
                "sha256:8e098148dd37b4ce3baca71fb394c81dc5d9c7728c95df695d2dca218edf40e6",
                "sha256:94aea8eff76ee6d1cdacb07dd2123a68283cb5569e0250feab1240058f53b623",
                "sha256:95eb302ff792e12aba9a8b8f8474ab229a83c103d74a750ec0bd1c1eea32e669",
                "sha256:9bd9b3b31adcb054116447ea22caa61a285d92e94d710aa5ec97992ff5eb7cf3",
                "sha256:9e608aafdb55eb9f255034709e20d5a83b6d60c054df0802fa9c9883d0a937aa",
                "sha256:a103b3a7069b62f5d4890ae1b8f0597618f628b286b03d4bc9195230b154bfa9",
                "sha256:a386ebe437176aab38c041de1260cd3ea459c6ce5263594399880bbc398225b2",
                "sha256:a38856a971c602f98472050165cea2cdc97709240373041b69030be15047691f",
                "sha256:a401b4598e5d3f4a9a811f3daf42ee2291790c7f9d74b18d75d6e21dda98a1a1",
                "sha256:a7647ebdfb9682b7bb97e2a5e7cb6ae735b1c25008a70b906aecca294ee96cf4",
                "sha256:aaf63899c94de41fe3cf934601b0f7ccb6b428c6e4eeb80da72c58eab077b19a",
                "sha256:b0dac0ff919ba34d4df1b6131f59ce95b08b9065233446be7e459f95554c0dc8",
                "sha256:baacc6aee0b2ef6f3d308e197b5d7a81c0e70b06beae1f1fcacffdbd124fe0e3",
                "sha256:bf420121d4c8dce6b889f0e8e4ec0ca34b7f40186203f06a946fa0276ba54029",
                "sha256:c04a46716adde8d927adb9457bbe39cf473e1e2c2f5d0a16ceb837e5d841ad4f",
                "sha256:c0b21078a4b56965e2b12f247467b234734491897e99c1d51cee628da9786959",
                "sha256:c1c76a1743432b4b60ab3358c937a3fe1341c828ae6194108a94c69028247f22",
                "sha256:c4983bf937209c57240cff65906b18bb35e64ae872da6a0db937d7b4af845dd7",
                "sha256:c4fb39a81950ec280984b3a44f5bd12819953dc5fa3a7e6fa7a80db5ee853952",
                "sha256:c57921cda3a80d0f2b8aec7e25c8aa14479ea92b5b51b6876d975d925a2ea346",
                "sha256:c8063cf17b19661471ecbdb3df1c84f24ad2e389e326ccaf89e3fb2484d8dd7e",
                "sha256:ccd16eb18a849fd8dcb23e23380e2f0a354e8daa0c984b8a732d9cfaba3a776d",
                "sha256:cd6dbe0238f7743d0efe563ab46294f54f9bc8f4b9bcf57c3c666cc5bc9d1299",
                "sha256:d62e51710986674142526ab9f78663ca2b0726066ae26b78b22e0f5e571238dd",
                "sha256:db901e2ac34c931d73054d9797383d0f8009991e723dab15109740a63e7f902a",
                "sha256:e03b8895a6990c9ab2cdcd0f2fe44088ca1c65ae592b8f795c3294af00a461c3",
                "sha256:e1c8a2f4c69e08e89632defbfabec2feb8a8d99edc9f89ce33c4b9e36ab63037",
                "sha256:e4b749b9cc6ee664a3300bb3a273c1ca8068c46be705b6c31cf5d276f8628a94",
                "sha256:e6a5bf2cba5ae1bb80b154ed68a3cfa2fa00fde979a7f50d6598d3e17d9ac20c",
                "sha256:e857a2232ba53ae940d3456f7533ce6ca98b81917d47adc3c7fd55dad8fab858",
                "sha256:ee4006268ed33370957f55bf2e6f4d263eaf4dc3cfc473d1d90baff6ed36ce4a",
                "sha256:eef9df1eefada2c09a5e7a40991b9fc6ac6ef20b1372abd48d2794a316dc0449",
                "sha256:f058f6963fd82eb143c692cecdc89e075fa0828db2e5b291070485390b2f1c9c",
                "sha256:f25c229a6ba38a35ae6e25ca1264621cc25d4d38dca2942a7fce0b67a4efe918",
                "sha256:f2a1d0fd4242bd8643ce6f98927cf9c04540af6efa92323e9d3124f57727bfc1",
                "sha256:f7560358a6811e52e9c4d142d497f1a6e10103d3a6881f18d04dbce3729c0e2c",
                "sha256:f779d3ad205f108d14e99bb3859aa7dd8e9c68874617c72354d7ecaec2a054ac",
                "sha256:f87f746ee241d30d6ed93969de31e5ffd09a2961a051e60ae6bddde9ec3583aa"
            ],
            "markers": "python_full_version >= '3.7.0' and sys_platform == 'linux'",
            "version": "==3.2.0"
        },
        "idna": {
            "hashes": [
                "sha256:814f528e8dead7d329833b91c5faa87d60bf71824cd12a7530b5526063d02cb4",
                "sha256:90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2"
            ],
            "markers": "python_version >= '3.5' and sys_platform == 'linux'",
            "version": "==3.4"
        },
        "requests": {
            "hashes": [
                "sha256:58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f",
                "sha256:942c5a758f98d790eaed1a29cb6eefc7ffb0d1cf7af05c3d2791656dbd6ad1e1"
            ],
            "markers": "python_version >= '3.7' and sys_platform == 'linux'",
            "version": "==2.31.0"
        },
        "urllib3": {
            "hashes": [
                "sha256:8d22f86aae8ef5e410d4f539fde9ce6b2113a001bb4d189e0aed70642d602b11",
                "sha256:de7df1803967d2c2a98e4b11bb7d6bd9210474c46e8a0401514e3a42a75ebde4"
            ],
            "index": "pypi",
            "markers": "python_version >= '3.7'",
            "version": "==2.0.4"
        }
    },
    "develop": {}
}

`

kolibri91 commented 10 months ago

There are on-going actions of pipenv: https://github.com/pypa/pipenv/issues/5561#issuecomment-1699012985

scastria commented 1 week ago

The started off using pipenv 2022.9.21 and used the --skip-lock option which defeated one of the main reasons to use pipenv. So I just recently upgraded to pipenv 2024.0.1 and am trying to use the lock files as best practice by checking them into my source repo. But if lock files are not cross platform, they are no good. I have the same problem as everyone else, develop on M1 mac or Windows, deploy to linux in cloud. It has been almost 1 year since this thread has been updated. What is the current status of this issue?

nk9 commented 1 week ago

I'd love to hear more about this being solved by pipenv itself. In the meantime, see the last approach here with “packages categories” which may help you work around the problem.

https://github.com/pypa/pipenv/issues/5723#issuecomment-1580834717

matteius commented 1 week ago

It is a challenging problem to solve--would love to support a reasonable solution without just going back to extra-index-urls which was the source of the disputed SEV vulnerability regarding package confusion attacks, the very thing I spent a lot of time considering how to prevent in pipenv which is how we end up here. I don't have a ton of excess time currently to devote to a multi-platform solution but I am supportive of finding one with the community. I think it would take some serious contributions from someone to get it across the finish line.

scastria commented 1 week ago

My current lazy plan is to NOT commit the lock files into source control and execute the lock during the CI build. I guess that is essentially the same as using --skip-lock. Then I will pin certain versions of dependencies to block them from auto-updating within the Pipfile itself as opposed to relying on the lock file to pin everything. I realize this is not best practice, but it is no worse than what I was doing with version 2022.9.21 and --skip-lock.

Reading through this thread, my favorite option was the idea to specify which platforms the Pipfile should support. This would tell pipenv which platforms to lock. I don't have a lot of internal knowledge on how Pypi works, but docker is able to pull and run images for a platform different than the host by specifying a --platform switch. I am assuming this user specified platform is used in the API queries made to docker hub to pull down available images. I would have assumed that the code that reads all the PyPI indices (public and private) includes the platform allowing pipenv to run the dependency mechanism multiple times for each platform listed in the Pipfile.

matteius commented 1 week ago

would have assumed that the code that reads all the PyPI indices (public and private) includes the platform allowing pipenv to run the dependency mechanism multiple times for each platform listed in the Pipfile.

I think that is what it does do, except for packages that don't provide pre-built wheels, that is when multi-platform is really problematic because you typically cannot build the sdist for the alternative OS to obtain the hashes when locking. I believe that is why some have proposed lock file sections, or overrides or even separate files specific to platforms. In theory something like that may be tenable, but its a complex landscape that someone needs to dig deeply into to improve the conditions.

scastria commented 1 week ago

Are the hashes just to prevent tampering? If so, I would be happy with a lock file that does not include hashes and instead just records the versions of everything. This would at least prevent a dependency from auto-upgrading when a new release comes out, like NumPy 2.x did very recently and broke our builds. Would a simplified lock file with just versions and no hashes make the cross platform problem easier to solve?

matteius commented 1 week ago

@scastria that is correct --- One thing to maybe tryin the interim is have the pipeline generate the requirements file from the lock file you use locally (possibly with hashes) and see if your CI can install with just pip install -r requirements.txt which would be quite similar to what you are describing.

pypa / pipenv

Multi Platform support for Pipfile.lock #5130

Is your feature request related to a problem? Please describe.

Describe the solution you'd like