search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.83k stars 1.87k forks source link

Pipfile sections are sorted before adding packages with extras #6171

Closed aradkdj closed 1 week ago

aradkdj commented 3 months ago

Issue description

When sort_pipfile is set to ture, if you install a new package with extras, it gets put at the end of the respective section instead of being sorted. If you previously had a package with extras at the end of the section, that one will get sorted but the new package with extras you're installing will still be put at the end of the section.

Expected result

The recently installed package gets sorted within its section.

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[pipenv]
sort_pipfile = true

[packages]
asgiref = "*"
psycopg = {extras = ["c"], version = "*"}
zenpy = "*"

[dev-packages]

[requires]
python_version = "3.10"
python_full_version = "3.10.14"

Actual result

The recently installed package gets put at the end of the section.

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[pipenv]
sort_pipfile = true

[packages]
asgiref = "*"
zenpy = "*"
psycopg = {extras = ["c"], version = "*"}

[dev-packages]

[requires]
python_version = "3.10"
python_full_version = "3.10.14"

Steps to replicate

  1. Start with a sorted Pipfile such as 
    
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[pipenv] sort_pipfile = true

[packages] asgiref = "" zenpy = ""

[dev-packages]

[requires] python_version = "3.10" python_full_version = "3.10.14"

2. Add a new package with extras (`pipenv install psycopg[c]`). It will get put at the end of the section instead of between asgiref and zenpy.

<details><summary>Pipfile</summary>

```toml
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[pipenv]
sort_pipfile = true

[packages]
asgiref = "*"
zenpy = "*"
psycopg = {extras = ["c"], version = "*"}

[dev-packages]

[requires]
python_version = "3.10"
python_full_version = "3.10.14"

  1. (Optional) add another package with extras (pipenv install sqlalchemy[async]). The previous package with extras will be sorted and this new one will be put at the end of the section.
Pipfile ```toml [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi" [pipenv] sort_pipfile = true [packages] asgiref = "*" psycopg = {extras = ["c"], version = "*"} zenpy = "*" sqlalchemy = {extras = ["async"], version = "*"} [dev-packages] [requires] python_version = "3.10" python_full_version = "3.10.14" ```
$ pipenv --support Pipenv version: `'2023.12.1'` Pipenv location: `'/home/user/.local/lib/python3.12/site-packages/pipenv'` Python location: `'/usr/bin/python3'` OS Name: `'posix'` User pip version: `'23.3.2'` user Python installations found: PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.12.3', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_python_implementation': 'CPython', 'platform_release': '6.8.11-300.fc40.x86_64', 'platform_system': 'Linux', 'platform_version': '#1 SMP PREEMPT_DYNAMIC Mon May 27 14:53:33 UTC 2024', 'python_full_version': '3.12.3', 'python_version': '3.12', 'sys_platform': 'linux'} ``` System environment variables: - `SHELL` - `SESSION_MANAGER` - `SNAP_REVISION` - `PYENV_SHELL` - `HISTCONTROL` - `XDG_MENU_PREFIX` - `SNAP_REAL_HOME` - `TERMINAL_EMULATOR` - `SNAP_USER_COMMON` - `HOSTNAME` - `HISTSIZE` - `SSH_AUTH_SOCK` - `MEMORY_PRESSURE_WRITE` - `TERM_SESSION_ID` - `SNAP_INSTANCE_KEY` - `XMODIFIERS` - `DESKTOP_SESSION` - `BAMF_DESKTOP_FILE_HINT` - `EDITOR` - `SNAP_EUID` - `PWD` - `XDG_SESSION_DESKTOP` - `LOGNAME` - `XDG_SESSION_TYPE` - `SYSTEMD_EXEC_PID` - `XAUTHORITY` - `DESKTOP_STARTUP_ID` - `SDL_VIDEO_MINIMIZE_ON_FOCUS_LOSS` - `SNAP_CONTEXT` - `GJS_DEBUG_TOPICS` - `GDM_LANG` - `HOME` - `USERNAME` - `LANG` - `LS_COLORS` - `XDG_CURRENT_DESKTOP` - `VIRTUAL_ENV` - `MEMORY_PRESSURE_WATCH` - `WAYLAND_DISPLAY` - `SNAP_ARCH` - `SNAP_INSTANCE_NAME` - `SNAP_USER_DATA` - `INVOCATION_ID` - `MANAGERPID` - `SNAP_REEXEC` - `SNAP_UID` - `STEAM_FRAME_FORCE_CLOSE` - `GJS_DEBUG_OUTPUT` - `GNOME_SETUP_DISPLAY` - `XDG_ACTIVATION_TOKEN` - `XDG_SESSION_CLASS` - `TERM` - `LESSOPEN` - `USER` - `SNAP` - `SNAP_COMMON` - `SNAP_VERSION` - `DISPLAY` - `SHLVL` - `SNAP_LIBRARY_PATH` - `SNAP_COOKIE` - `QT_IM_MODULE` - `VIRTUAL_ENV_PROMPT` - `SNAP_DATA` - `XDG_RUNTIME_DIR` - `PS1` - `PYENV_ROOT` - `DEBUGINFOD_URLS` - `SNAP_NAME` - `JOURNAL_STREAM` - `XDG_DATA_DIRS` - `PATH` - `GDMSESSION` - `DBUS_SESSION_BUS_ADDRESS` - `MAIL` - `GIO_LAUNCHED_DESKTOP_FILE_PID` - `GIO_LAUNCHED_DESKTOP_FILE` - `_` - `PIP_DISABLE_PIP_VERSION_CHECK` - `PYTHONDONTWRITEBYTECODE` - `PYTHONFINDER_IGNORE_UNSUPPORTED` Pipenv–specific environment variables: Debug–specific environment variables: - `PATH`: `/home/user/.pyenv/versions/3.10.14/bin:/home/user/.local/share/virtualenvs/pythonProject-bFLmJOh0/bin:/home/user/.pyenv/shims:/home/user/.pyenv/bin:/home/user/.local/bin:/home/user/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/var/lib/snapd/snap/bin` - `SHELL`: `/bin/bash` - `EDITOR`: `/usr/bin/nano` - `LANG`: `en_US.UTF-8` - `PWD`: `/home/user/Documents/learning/pythonProject` - `VIRTUAL_ENV`: `/home/user/.local/share/virtualenvs/pythonProject-bFLmJOh0` --------------------------- Contents of `Pipfile` ('/home/user/Documents/learning/pythonProject/Pipfile'): ```toml [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi" [pipenv] sort_pipfile = true [packages] asgiref = "*" zenpy = "*" psycopg = {extras = ["c"], version = "*"} [dev-packages] [requires] python_version = "3.10" python_full_version = "3.10.14" ``` Contents of `Pipfile.lock` ('/home/user/Documents/learning/pythonProject/Pipfile.lock'): ```json { "_meta": { "hash": { "sha256": "f89b0e17c8d001893b901878be733a0705afc39a6cd04041a6ddffdfee693e5b" }, "pipfile-spec": 6, "requires": { "python_full_version": "3.10.14", "python_version": "3.10" }, "sources": [ { "name": "pypi", "url": "https://pypi.org/simple", "verify_ssl": true } ] }, "default": { "asgiref": { "hashes": [ "sha256:3e1e3ecc849832fe52ccf2cb6686b7a55f82bb1d6aee72a58826471390335e47", "sha256:c343bd80a0bec947a9860adb4c432ffa7db769836c64238fc34bdc3fec84d590" ], "index": "pypi", "markers": "python_version >= '3.8'", "version": "==3.8.1" }, "cachetools": { "hashes": [ "sha256:0abad1021d3f8325b2fc1d2e9c8b9c9d57b04c3932657a72465447332c24d945", "sha256:ba29e2dfa0b8b556606f097407ed1aa62080ee108ab0dc5ec9d6a723a007d105" ], "markers": "python_version >= '3.7'", "version": "==5.3.3" }, "certifi": { "hashes": [ "sha256:3cd43f1c6fa7dedc5899d69d3ad0398fd018ad1a17fba83ddaf78aa46c747516", "sha256:ddc6c8ce995e6987e7faf5e3f1b02b302836a0e5d98ece18392cb1a36c72ad56" ], "markers": "python_version >= '3.6'", "version": "==2024.6.2" }, "charset-normalizer": { "hashes": [ "sha256:06435b539f889b1f6f4ac1758871aae42dc3a8c0e24ac9e60c2384973ad73027", "sha256:06a81e93cd441c56a9b65d8e1d043daeb97a3d0856d177d5c90ba85acb3db087", "sha256:0a55554a2fa0d408816b3b5cedf0045f4b8e1a6065aec45849de2d6f3f8e9786", "sha256:0b2b64d2bb6d3fb9112bafa732def486049e63de9618b5843bcdd081d8144cd8", "sha256:10955842570876604d404661fbccbc9c7e684caf432c09c715ec38fbae45ae09", "sha256:122c7fa62b130ed55f8f285bfd56d5f4b4a5b503609d181f9ad85e55c89f4185", "sha256:1ceae2f17a9c33cb48e3263960dc5fc8005351ee19db217e9b1bb15d28c02574", "sha256:1d3193f4a680c64b4b6a9115943538edb896edc190f0b222e73761716519268e", "sha256:1f79682fbe303db92bc2b1136016a38a42e835d932bab5b3b1bfcfbf0640e519", "sha256:2127566c664442652f024c837091890cb1942c30937add288223dc895793f898", "sha256:22afcb9f253dac0696b5a4be4a1c0f8762f8239e21b99680099abd9b2b1b2269", "sha256:25baf083bf6f6b341f4121c2f3c548875ee6f5339300e08be3f2b2ba1721cdd3", "sha256:2e81c7b9c8979ce92ed306c249d46894776a909505d8f5a4ba55b14206e3222f", "sha256:3287761bc4ee9e33561a7e058c72ac0938c4f57fe49a09eae428fd88aafe7bb6", "sha256:34d1c8da1e78d2e001f363791c98a272bb734000fcef47a491c1e3b0505657a8", "sha256:37e55c8e51c236f95b033f6fb391d7d7970ba5fe7ff453dad675e88cf303377a", "sha256:3d47fa203a7bd9c5b6cee4736ee84ca03b8ef23193c0d1ca99b5089f72645c73", "sha256:3e4d1f6587322d2788836a99c69062fbb091331ec940e02d12d179c1d53e25fc", "sha256:42cb296636fcc8b0644486d15c12376cb9fa75443e00fb25de0b8602e64c1714", "sha256:45485e01ff4d3630ec0d9617310448a8702f70e9c01906b0d0118bdf9d124cf2", "sha256:4a78b2b446bd7c934f5dcedc588903fb2f5eec172f3d29e52a9096a43722adfc", "sha256:4ab2fe47fae9e0f9dee8c04187ce5d09f48eabe611be8259444906793ab7cbce", "sha256:4d0d1650369165a14e14e1e47b372cfcb31d6ab44e6e33cb2d4e57265290044d", "sha256:549a3a73da901d5bc3ce8d24e0600d1fa85524c10287f6004fbab87672bf3e1e", "sha256:55086ee1064215781fff39a1af09518bc9255b50d6333f2e4c74ca09fac6a8f6", "sha256:572c3763a264ba47b3cf708a44ce965d98555f618ca42c926a9c1616d8f34269", "sha256:573f6eac48f4769d667c4442081b1794f52919e7edada77495aaed9236d13a96", "sha256:5b4c145409bef602a690e7cfad0a15a55c13320ff7a3ad7ca59c13bb8ba4d45d", "sha256:6463effa3186ea09411d50efc7d85360b38d5f09b870c48e4600f63af490e56a", "sha256:65f6f63034100ead094b8744b3b97965785388f308a64cf8d7c34f2f2e5be0c4", "sha256:663946639d296df6a2bb2aa51b60a2454ca1cb29835324c640dafb5ff2131a77", "sha256:6897af51655e3691ff853668779c7bad41579facacf5fd7253b0133308cf000d", "sha256:68d1f8a9e9e37c1223b656399be5d6b448dea850bed7d0f87a8311f1ff3dabb0", "sha256:6ac7ffc7ad6d040517be39eb591cac5ff87416c2537df6ba3cba3bae290c0fed", "sha256:6b3251890fff30ee142c44144871185dbe13b11bab478a88887a639655be1068", "sha256:6c4caeef8fa63d06bd437cd4bdcf3ffefe6738fb1b25951440d80dc7df8c03ac", "sha256:6ef1d82a3af9d3eecdba2321dc1b3c238245d890843e040e41e470ffa64c3e25", "sha256:753f10e867343b4511128c6ed8c82f7bec3bd026875576dfd88483c5c73b2fd8", "sha256:7cd13a2e3ddeed6913a65e66e94b51d80a041145a026c27e6bb76c31a853c6ab", "sha256:7ed9e526742851e8d5cc9e6cf41427dfc6068d4f5a3bb03659444b4cabf6bc26", "sha256:7f04c839ed0b6b98b1a7501a002144b76c18fb1c1850c8b98d458ac269e26ed2", "sha256:802fe99cca7457642125a8a88a084cef28ff0cf9407060f7b93dca5aa25480db", "sha256:80402cd6ee291dcb72644d6eac93785fe2c8b9cb30893c1af5b8fdd753b9d40f", "sha256:8465322196c8b4d7ab6d1e049e4c5cb460d0394da4a27d23cc242fbf0034b6b5", "sha256:86216b5cee4b06df986d214f664305142d9c76df9b6512be2738aa72a2048f99", "sha256:87d1351268731db79e0f8e745d92493ee2841c974128ef629dc518b937d9194c", "sha256:8bdb58ff7ba23002a4c5808d608e4e6c687175724f54a5dade5fa8c67b604e4d", "sha256:8c622a5fe39a48f78944a87d4fb8a53ee07344641b0562c540d840748571b811", "sha256:8d756e44e94489e49571086ef83b2bb8ce311e730092d2c34ca8f7d925cb20aa", "sha256:8f4a014bc36d3c57402e2977dada34f9c12300af536839dc38c0beab8878f38a", "sha256:9063e24fdb1e498ab71cb7419e24622516c4a04476b17a2dab57e8baa30d6e03", "sha256:90d558489962fd4918143277a773316e56c72da56ec7aa3dc3dbbe20fdfed15b", "sha256:923c0c831b7cfcb071580d3f46c4baf50f174be571576556269530f4bbd79d04", "sha256:95f2a5796329323b8f0512e09dbb7a1860c46a39da62ecb2324f116fa8fdc85c", "sha256:96b02a3dc4381e5494fad39be677abcb5e6634bf7b4fa83a6dd3112607547001", "sha256:9f96df6923e21816da7e0ad3fd47dd8f94b2a5ce594e00677c0013018b813458", "sha256:a10af20b82360ab00827f916a6058451b723b4e65030c5a18577c8b2de5b3389", "sha256:a50aebfa173e157099939b17f18600f72f84eed3049e743b68ad15bd69b6bf99", "sha256:a981a536974bbc7a512cf44ed14938cf01030a99e9b3a06dd59578882f06f985", "sha256:a9a8e9031d613fd2009c182b69c7b2c1ef8239a0efb1df3f7c8da66d5dd3d537", "sha256:ae5f4161f18c61806f411a13b0310bea87f987c7d2ecdbdaad0e94eb2e404238", "sha256:aed38f6e4fb3f5d6bf81bfa990a07806be9d83cf7bacef998ab1a9bd660a581f", "sha256:b01b88d45a6fcb69667cd6d2f7a9aeb4bf53760d7fc536bf679ec94fe9f3ff3d", "sha256:b261ccdec7821281dade748d088bb6e9b69e6d15b30652b74cbbac25e280b796", "sha256:b2b0a0c0517616b6869869f8c581d4eb2dd83a4d79e0ebcb7d373ef9956aeb0a", "sha256:b4a23f61ce87adf89be746c8a8974fe1c823c891d8f86eb218bb957c924bb143", "sha256:bd8f7df7d12c2db9fab40bdd87a7c09b1530128315d047a086fa3ae3435cb3a8", "sha256:beb58fe5cdb101e3a055192ac291b7a21e3b7ef4f67fa1d74e331a7f2124341c", "sha256:c002b4ffc0be611f0d9da932eb0f704fe2602a9a949d1f738e4c34c75b0863d5", "sha256:c083af607d2515612056a31f0a8d9e0fcb5876b7bfc0abad3ecd275bc4ebc2d5", "sha256:c180f51afb394e165eafe4ac2936a14bee3eb10debc9d9e4db8958fe36afe711", "sha256:c235ebd9baae02f1b77bcea61bce332cb4331dc3617d254df3323aa01ab47bd4", "sha256:cd70574b12bb8a4d2aaa0094515df2463cb429d8536cfb6c7ce983246983e5a6", "sha256:d0eccceffcb53201b5bfebb52600a5fb483a20b61da9dbc885f8b103cbe7598c", "sha256:d965bba47ddeec8cd560687584e88cf699fd28f192ceb452d1d7ee807c5597b7", "sha256:db364eca23f876da6f9e16c9da0df51aa4f104a972735574842618b8c6d999d4", "sha256:ddbb2551d7e0102e7252db79ba445cdab71b26640817ab1e3e3648dad515003b", "sha256:deb6be0ac38ece9ba87dea880e438f25ca3eddfac8b002a2ec3d9183a454e8ae", "sha256:e06ed3eb3218bc64786f7db41917d4e686cc4856944f53d5bdf83a6884432e12", "sha256:e27ad930a842b4c5eb8ac0016b0a54f5aebbe679340c26101df33424142c143c", "sha256:e537484df0d8f426ce2afb2d0f8e1c3d0b114b83f8850e5f2fbea0e797bd82ae", "sha256:eb00ed941194665c332bf8e078baf037d6c35d7c4f3102ea2d4f16ca94a26dc8", "sha256:eb6904c354526e758fda7167b33005998fb68c46fbc10e013ca97f21ca5c8887", "sha256:eb8821e09e916165e160797a6c17edda0679379a4be5c716c260e836e122f54b", "sha256:efcb3f6676480691518c177e3b465bcddf57cea040302f9f4e6e191af91174d4", "sha256:f27273b60488abe721a075bcca6d7f3964f9f6f067c8c4c605743023d7d3944f", "sha256:f30c3cb33b24454a82faecaf01b19c18562b1e89558fb6c56de4d9118a032fd5", "sha256:fb69256e180cb6c8a894fee62b3afebae785babc1ee98b81cdf68bbca1987f33", "sha256:fd1abc0d89e30cc4e02e4064dc67fcc51bd941eb395c502aac3ec19fab46b519", "sha256:ff8fa367d09b717b2a17a052544193ad76cd49979c805768879cb63d9ca50561" ], "markers": "python_full_version >= '3.7.0'", "version": "==3.3.2" }, "idna": { "hashes": [ "sha256:028ff3aadf0609c1fd278d8ea3089299412a7a8b9bd005dd08b9f8285bcb5cfc", "sha256:82fee1fc78add43492d3a1898bfa6d8a904cc97d8427f683ed8e798d07761aa0" ], "markers": "python_version >= '3.5'", "version": "==3.7" }, "psycopg": { "extras": [ "c" ], "hashes": [ "sha256:92d7b78ad82426cdcf1a0440678209faa890c6e1721361c2f8901f0dccd62961", "sha256:dca5e5521c859f6606686432ae1c94e8766d29cc91f2ee595378c510cc5b0731" ], "markers": "python_version >= '3.7'", "version": "==3.1.19" }, "psycopg-c": { "hashes": [ "sha256:8e90f53c430e7d661cb3a9298e2761847212ead1b24c5fb058fc9d0fd9616017" ], "version": "==3.1.19" }, "python-dateutil": { "hashes": [ "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427" ], "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'", "version": "==2.9.0.post0" }, "pytz": { "hashes": [ "sha256:2a29735ea9c18baf14b448846bde5a48030ed267578472d8955cd0e7443a9812", "sha256:328171f4e3623139da4983451950b28e95ac706e13f3f2630a879749e7a8b319" ], "version": "==2024.1" }, "requests": { "hashes": [ "sha256:55365417734eb18255590a9ff9eb97e9e1da868d4ccd6402399eaf68af20a760", "sha256:70761cfe03c773ceb22aa2f671b4757976145175cdfca038c02654d061d6dcc6" ], "markers": "python_version >= '3.8'", "version": "==2.32.3" }, "six": { "hashes": [ "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926", "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254" ], "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'", "version": "==1.16.0" }, "typing-extensions": { "hashes": [ "sha256:6024b58b69089e5a89c347397254e35f1bf02a907728ec7fee9bf0fe837d203a", "sha256:915f5e35ff76f56588223f15fdd5938f9a1cf9195c0de25130c627e4d597f6d1" ], "markers": "python_version >= '3.8'", "version": "==4.12.1" }, "urllib3": { "hashes": [ "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" ], "markers": "python_version >= '3.8'", "version": "==2.2.1" }, "zenpy": { "hashes": [ "sha256:39303ce90a5cf8ed71e753fa5f0ae429ad3e52324e633c57e62cd8f987e3ef3f" ], "index": "pypi", "version": "==2.0.49" } }, "develop": {} } ```
oz123 commented 3 months ago

Is this a bug report or a feature request?

matteius commented 3 months ago

I believe its a bug report with the sort_pipfile = true feature and dependencies with extras

aradkdj commented 3 months ago

It's a bug report, I already have a (currently failing) test case to validate it. I also could try and tackle it

sanspareilsmyn commented 2 months ago

@aradkdj @matteius @oz123 I had a dive on this issue and encountered several problems.

  1. In pipenv/project.py, there's a logic like this. This handles sort_pipfile flag.

    def _sort_category(self, category):
    # toml tables won't maintain sorted dictionary order
    # so construct the table in the order that we need
    table = tomlkit.table()
    for key, value in sorted(category.items()):
        table.add(key, value)

    return table

  2. tomlkit.table() has its own appending logic (append and _raw_append), which is implemented in vendor/tomlkit/container.py.

  3. No matter how we sort the order before adding to tomlkit.table(), table.add() ruins sorted order.

  4. I had some troubles figuring out which logic blocks us from sorting package keys in alphabetical orders.

oz123 commented 1 week ago

@Naofal-Helal thank you for your contribution fixing this issue.