pip-compile gives different result on macOS and Debian

[Opalo]

Environment Versions

1. OS

macOS:

~ ❯ sw_vers                                                                                                                                                                                                                                
ProductName:        macOS
ProductVersion:     13.6
BuildVersion:       22G120
~ ❯

and: python:3.9.16-slim docker image

root@726959d3b1ba:/# cat /etc/issue
Debian GNU/Linux 11 \n \l

2. Python

macOS:

/tmp ❯ python -V                                                                                                                                                                                                                               
Python 3.9.16

and: python:3.9.16-slim docker image

root@bdc7b5caa828:/# python -V
Python 3.9.16

3. PIP

macOS:

/tmp ❯ pip -V                                                                                                                                                                                                                                  
pip 23.3.1 from /private/tmp/.venv/lib/python3.9/site-packages/pip (python 3.9)

and: python:3.9.16-slim docker image

root@bdc7b5caa828:/# pip -V
pip 23.3.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)

4. pip-tools

macOS:

/tmp ❯ pip-compile --version                                                                                                                                                                                                                  
pip-compile, version 7.3.0

and: python:3.9.16-slim docker image

root@bdc7b5caa828:/# pip-compile --version
pip-compile, version 7.3.0

Steps to replicate

1. Use python 3.9.16
2. Use pip 23.3.1
3. Use the following requirements.in file:

   jira==3.0.1
   sqlalchemy==1.4.48

4. Run pip-compile -o requirements.txt --strip-extras requirements.in on both OSes
5. Compare both generated requirements.txt

Expected result

Both generated requitements.txt should contain the same packages and versions.

Actual result

For macOS:

#
# This file is autogenerated by pip-compile with Python 3.9
# by the following command:
#
#    pip-compile --output-file=requirements.txt requirements.in
#
certifi==2023.7.22
    # via requests
charset-normalizer==3.3.1
    # via requests
defusedxml==0.7.1
    # via jira
idna==3.4
    # via requests
importlib-metadata==6.8.0
    # via keyring
jaraco-classes==3.3.0
    # via keyring
jira==3.0.1
    # via -r requirements.in
keyring==24.2.0
    # via jira
more-itertools==10.1.0
    # via jaraco-classes
oauthlib==3.2.2
    # via requests-oauthlib
requests==2.31.0
    # via
    #   jira
    #   requests-oauthlib
    #   requests-toolbelt
requests-oauthlib==1.3.1
    # via jira
requests-toolbelt==1.0.0
    # via jira
sqlalchemy==1.4.48
    # via -r requirements.in
urllib3==2.0.7
    # via requests
zipp==3.17.0
    # via importlib-metadata

# The following packages are considered to be unsafe in a requirements file:
# setuptools

For python:3.9.16-slim docker image:

#
# This file is autogenerated by pip-compile with Python 3.9
# by the following command:
#
#    pip-compile --output-file=requirements.txt requirements.in
#
certifi==2023.7.22
    # via requests
cffi==1.16.0
    # via cryptography
charset-normalizer==3.3.1
    # via requests
cryptography==41.0.5
    # via secretstorage
defusedxml==0.7.1
    # via jira
greenlet==3.0.0
    # via sqlalchemy
idna==3.4
    # via requests
importlib-metadata==6.8.0
    # via keyring
jaraco-classes==3.3.0
    # via keyring
jeepney==0.8.0
    # via
    #   keyring
    #   secretstorage
jira==3.0.1
    # via -r requirements.in
keyring==24.2.0
    # via jira
more-itertools==10.1.0
    # via jaraco-classes
oauthlib==3.2.2
    # via requests-oauthlib
pycparser==2.21
    # via cffi
requests==2.31.0
    # via
    #   jira
    #   requests-oauthlib
    #   requests-toolbelt
requests-oauthlib==1.3.1
    # via jira
requests-toolbelt==1.0.0
    # via jira
secretstorage==3.3.3
    # via keyring
sqlalchemy==1.4.48
    # via -r requirements.in
urllib3==2.0.7
    # via requests
zipp==3.17.0
    # via importlib-metadata

# The following packages are considered to be unsafe in a requirements file:
# setuptools

Diff:

8a9,10
> cffi==1.16.0
>     # via cryptography
10a13,14
> cryptography==41.0.5
>     # via secretstorage
12a17,18
> greenlet==3.0.0
>     # via sqlalchemy
18a25,28
> jeepney==0.8.0
>     # via
>     #   keyring
>     #   secretstorage
26a37,38
> pycparser==2.21
>     # via cffi
35a48,49
> secretstorage==3.3.3
>     # via keyring

I don't have an idea why the generated files are different but I'd expect them to be the same. The problem started when I added the generated files (on macOS) to source control and it failed in GH action on GH CI. For the very beginning I thought it is related to python and pip versions - when I unified them (3.9.16 vs 3.9.18 for python) some differences disappeared. Then I thought pre-commit is to be blamed, finally it turned out that different requirements.txt files are generated even with the same versions of tools being used. The only difference is the OS.

[AndydeCleyre]

This is expected, as packages can specify different dependencies according to the current platform or Python version. Notice that keyring specifies a bunch of dependencies this way.

[Opalo]

Thanks @AndydeCleyre! What is the suggested course of action here then? Should I wrap pip-compile invocation in some docker container or can handle it differently? I need to have consistent results.

[webknjaz]

@Opalo it depends on your use-case. If it's an app deployed to a single well-defined environment, you might want to make sure you use the same env to manage the logfiles. If you have a library and test against a matrix of envs, you may opt to have a bunch of constraint files per each combo of Python, OS and arch. The latter would require some wrapper script to select different constraint files in different envs, I'm experimenting with such an approach @ https://github.com/cherrypy/cheroot/tree/main/requirements for quite a while and I like how it allows me to keep the CI green with no surprise transitive dependency bumps and transparent package updates.

[webknjaz]

Oh, and make sure to study the docs carefully. I think they can provide you with more insight.

[Opalo]

Thanks for all the hints! I've wrapped the generation into a docker container. And it all works fine.