search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.83k stars 1.87k forks source link

Exception when last block in Pipfile is a [[source]] and no trailing newline at end of file #6244

Closed gdiepen closed 12 hours ago

gdiepen commented 4 days ago

Issue description

During a pipenv install pyodbc with a Pipfile that has multiple sources it seems pipenv reorganizes the Pipfile to have all [[sources]] blocks at the top.

However, this reordering will fail if the last line is missing a trailing new-line

If you use the following Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = false
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.11"

[[source]]
url = "https://pypi.org/simple"
verify_ssl = false
name = "pypi2"

(note that I am using twice pypi.org, in our case this would be a private registry). Note that this Pipfile should NOT have a trailing newline after the last name = "pypi2" line!

Expected result

Using the above mentioned Pipfile in the statement

`pipenv install --verbose --extra-pip-args "--dry-run" pyodbc

should just work without a problem

Actual result

Executing the statement `pipenv install --verbose --extra-pip-args "--dry-run" pyodbc

with the above mentioned Pipfile results in the following error:

pipenv install --verbose --extra-pip-args "--dry-run" pyodbc
Requirements file provided! Importing into Pipfile...
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pipenv/project.py", line 700, in _parse_pipfile
    return tomlkit.parse(contents)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/api.py", line 86, in parse
    return Parser(string).parse()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 154, in parse
    key, value = self._parse_table()
                 ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 1038, in _parse_table
    result = self._parse_aot(result, full_key)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 1079, in _parse_aot
    _, table = self._parse_table(name_first)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 1002, in _parse_table
    item = self._parse_item()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 238, in _parse_item
    return self._parse_key_value(True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 334, in _parse_key_value
    cws, comment, trail = self._parse_comment_trail()
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 280, in _parse_comment_trail
    raise self.parse_error(UnexpectedCharError, c)
pipenv.vendor.tomlkit.exceptions.UnexpectedCharError: Unexpected character: '[' at line 9 col 14

If you check the Pipfile after this command has finished, you will see that pipenv has modified the original contents:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = false
name = "pypi"

[[source]]
url = "https://pypi.org/simple"
verify_ssl = false
name = "pypi2"[packages]
pyodbc = "*"

[dev-packages]

[requires]
python_version = "3.11"

As you can see, the [packages] section is started right after the name = "pypi2" key-value line (that originally did not have the trailing new-line)

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

See the earlier mentioned Pipfile (note: make sure you remove any trailing newline, e.g. with command truncate -s -1 Pipfile) and command to execute.

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

$ pipenv --support Pipenv version: `'2024.0.2'` Pipenv location: `'/usr/local/lib/python3.11/site-packages/pipenv'` Python location: `'/usr/local/bin/python'` OS Name: `'posix'` User pip version: `'24.0'` user Python installations found: PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.11.10', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_python_implementation': 'CPython', 'platform_release': '5.15.133.1-microsoft-standard-WSL2', 'platform_system': 'Linux', 'platform_version': '#1 SMP Thu Oct 5 21:02:42 UTC 2023', 'python_full_version': '3.11.10', 'python_version': '3.11', 'sys_platform': 'linux'} ``` System environment variables: - `PYTHONUNBUFFERED` - `HOSTNAME` - `PYTHON_VERSION` - `PWD` - `PYTHON_SETUPTOOLS_VERSION` - `container` - `HOME` - `LANG` - `LS_COLORS` - `GPG_KEY` - `https_proxy` - `TERM` - `SHLVL` - `HTTPS_PROXY` - `HTTP_PROXY` - `PYTHON_PIP_VERSION` - `http_proxy` - `PYTHONDONTWRITEBYTECODE` - `PYTHON_GET_PIP_SHA256` - `PYTHON_GET_PIP_URL` - `PATH` - `_` - `PIP_DISABLE_PIP_VERSION_CHECK` - `PYTHONFINDER_IGNORE_UNSUPPORTED` Pipenv–specific environment variables: Debug–specific environment variables: - `PATH`: `/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin` - `LANG`: `C.UTF-8` - `PWD`: `/application` --------------------------- Contents of `Pipfile` ('/application/Pipfile'): ```toml [[source]] url = "https://pypi.org/simple" verify_ssl = false name = "pypi" [packages] [dev-packages] [requires] python_version = "3.11" [[source]] url = "https://pypi.org/simple" verify_ssl = false name = "pypi2" ``` Contents of `Pipfile.lock` ('/application/Pipfile.lock'): ```json { "_meta": { "hash": { "sha256": "f68fbcf7316b462719ef6a182db46870756b9b585113d062202aea794b67e3d1" }, "pipfile-spec": 6, "requires": { "python_version": "3.11" }, "sources": [ { "name": "pypi", "url": "https://pypi.org/simple", "verify_ssl": false }, { "name": "pypi2", "url": "https://pypi.org/simple", "verify_ssl": false } ] }, "default": { "pyodbc": { "hashes": [ "sha256:02fe9821711a2d14415eaeb4deab471d2c8b7034b107e524e414c0e133c42248", "sha256:1c5e0cb79222aad4b31a3602e39b242683c29c6221a16ed43f45f18fd0b73659", "sha256:218bb75d4bc67075529a65ce8ec7daeed1d83c33dd7410450fbf68d43d184d28", "sha256:29425e2d366e7f5828b76c7993f412a3db4f18bd5bcee00186c00b5a5965e205", "sha256:2cbdbd019756285dc44bc35238a3ed8dfaa454e8c8b2c3462f1710cfeebfb290", "sha256:33f0f1d7764cefef6f787936bd6359670828a6086be67518ab951f1f7f503cda", "sha256:33f4984af38872e7bdec78007a34e4d43ae72bf9d0bae3344e79d9d0db157c0e", "sha256:3602136a936bc0c1bb9722eb2fbf2042b3ff1ddccdc4688e514b82d4b831563b", "sha256:397feee44561a6580be08cedbe986436859563f4bb378f48224655c8e987ea60", "sha256:3c36448322f8d6479d87c528cf52401a6ea4f509b9637750b67340382b4e1b40", "sha256:406b8fa2133a7b6a713aa5187dba2d08cf763b5884606bed77610a7660fdfabe", "sha256:735f6da3762e5856b5580be0ed96bb946948346ebd1e526d5169a5513626a67a", "sha256:84df3bbce9bafe65abd25788d55c9f1da304f6115d70f25758ff8c85f3ce0517", "sha256:92caed9d445815ed3f7e5a1249e29a4600ebc1e99404df81b6ed7671074c9227", "sha256:96b2a8dc27693a517e3aad3944a7faa8be95d40d7ec1eda51a1885162eedfa33", "sha256:a1bd14633e91b7a9814f4fd944c9ebb89fb7f1fd4710c4e3999b5ef041536347", "sha256:a2bbd2e75c77dee9f3cd100c3246110abaeb9af3f7fa304ccc2934ff9c6a4fa4", "sha256:aa4e02d3a9bf819394510b726b25f1566f8b3f0891ca400ad2d4c8b86b535b78", "sha256:aa6f46377da303bf79bcb4b559899507df4b2559f30dcfdf191358ee4b99f3ab", "sha256:af5282cc8b667af97d76f4955250619a53f25486cbb6b1f45a06b781006ffa0b", "sha256:b0df69e3a500791b70b5748c68a79483b24428e4c16027b56aa0305e95c143a4", "sha256:b19d7f44cfee89901e482f554a88177e83fae76b03c3f830e0023a195d840220", "sha256:be3b1c36c31ec7d73d0b34a8ad8743573763fadd8f2bceef1e84408252b48dce", "sha256:bed1c843565d3a4fd8c332ebceaf33efe817657a0505eacb97dd1b786a985b0b", "sha256:c3b65343557f4c7753204e06f4c82c97ed212a636501f4bc27c5ce0e549eb3e8", "sha256:c5bb4e43f6c72f5fa2c634570e0d761767d8ea49f39205229b812fb4d3fe05aa", "sha256:d3d9cc4af703c4817b6e604315910b0cf5dcb68056d52b25ca072dd59c52dcbc", "sha256:e71a51c252b503b4d753e21ed31e640015fc0d00202d42ea42f2396fcc924b4a", "sha256:e738c5eedb4a0cbab20cc008882f49b106054499db56864057c2530ff208cf32", "sha256:eae576b3b67d21d6f237e18bb5f3df8323a2258f52c3e3afeef79269704072a9", "sha256:f8488c3818f12207650836c5c6f7352f9ff9f56a05a05512145995e497c0bbb1" ], "index": "pypi", "markers": "python_version >= '3.8'", "version": "==5.1.0" } }, "develop": {} } ```
oz123 commented 2 days ago

The exception is happening at the tomlkit parse. I wonder if there is a bug upstream for that.

What is also surprise is that we are loading pipfile via tomlkit and not via plette.

gdiepen commented 12 hours ago

Did some more debugging, and you are right, not a problem on the side of pipenv.

Created simple example that reproduces the problem and created an issue in the tomlkit repo: https://github.com/python-poetry/tomlkit/issues/381