matteius
commented
3 weeks ago Closed matteius closed 4 days ago
matteius
commented
3 weeks ago 
fredcohen-aa
commented
3 weeks ago Seeing what I assume is the the same issue with this dependency in latest pipenv:
tensorflow-cpu = {"version"="~=2.13.0", markers="platform_machine != 'arm64'"}
Installing dependencies from Pipfile.lock (611b50)...
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 36, in __init__
parsed = _parse_requirement(requirement_string)
File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/_parser.py", line 62, in parse_requirement
return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/_parser.py", line 80, in _parse_requirement
url, specifier, marker = _parse_requirement_details(tokenizer)
File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/_parser.py", line 124, in _parse_requirement_details
marker = _parse_requirement_marker(
File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/_parser.py", line 145, in _parse_requirement_marker
tokenizer.raise_syntax_error(
File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/_tokenizer.py", line 167, in raise_syntax_error
raise ParserSyntaxError(
pipenv.patched.pip._vendor.packaging._tokenizer.ParserSyntaxError: Expected end or semicolon (after name and no valid version specifier)
tensorflow==~=2.13.0
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 362, in _parse_req_string
return get_requirement(req_as_string)
File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_internal/utils/packaging.py", line 45, in get_requirement
return Requirement(req_string)
File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 38, in __init__
raise InvalidRequirement(str(e)) from e
pipenv.patched.pip._vendor.packaging.requirements.InvalidRequirement: Expected end or semicolon (after name and no valid version specifier)
tensorflow==~=2.13.0@fredcohen-aa 2024.0.3 is working better for vcs (assuming you aren't using env vars), but also what you are reporting is slightly different than a vcs issue though it seems to manifest similarly. I plan to look at this more this weekend, but if anyone has any ideas ahead of that, be sure to comment. We did upgrade a minor version of pip too, which is something else at play. I am fairly surprised the tests + the ones added didn't catch this.
matteius
commented
3 weeks ago I think this is going to be challenging issue as we move forward with pip >= 24.1.2 it appears to be more strict about some things such as version specifiers. For example -- I am unable to lock the parent package django-trench successfully from git/vcs, sometimes it did work but always had a null ref, and adding additional vcs it failed it. But I forked the package and tried changing the version like so: https://github.com/matteius/django-trench/commit/821b4215f1b0cb00f36943f056afbae0c2441bf3
Now it just works again 🤔
matteius
commented
3 weeks ago @fredcohen-aa please open a different issue report for what you are seeing with even more details than you provide above.
fredcohen-aa
commented
2 weeks ago @fredcohen-aa please open a different issue report for what you are seeing with even more details than you provide above.
Issue #6273 opened
matteius
commented
1 week ago Analysis for Issue #6266:
1. Problem Summary:
The core problem is that Pipenv's resolver in version 2024.1.0 mistakenly attempts to resolve VCS dependencies as regular dependencies, leading to a ParserSyntaxError. The issue seems to occur when processing requirement strings, specifically when parsing VCS URLs. The provided traceback indicates an issue with a requirement line containing "pyminizip==" without a valid version specifier, suggesting Pipenv is treating the VCS URL as a standard package name and version.
2. Comment Discussion Analysis:
Similar but Different Issue: The maintainer (@matteius) acknowledges the issue but points out that the reported case differs from a pure VCS issue. They mention that Pipenv 2024.0.3 improved VCS handling, implying there may be separate bugs related to VCS and marker parsing.
Stricter Parsing in Pip: The maintainer highlights the stricter parsing in newer Pip versions (>= 24.1.2) as a potential contributing factor. This suggests that Pipenv's internal logic for constructing requirement strings may need to adapt to these changes.
Potential Conflict with Markers: The comment from @fredcohen-aa suggests that the issue might be related to the presence of environment markers in the requirement line. The traceback he provides shows a similar error ("Expected end or semicolon") when using a marker alongside a version specifier.
3. Proposed Resolution:
The issue seems to be rooted in how Pipenv constructs requirement strings for VCS dependencies. Pipenv needs to ensure that VCS URLs are treated differently from standard package names and versions. This includes:
Preserving VCS URL structure: The requirement string passed to the resolver should retain the VCS prefix and the entire URL structure, including the "@" symbol, to ensure correct parsing.
Handling environment markers: If markers are present, they should be appended correctly to the VCS URL requirement string, potentially using a semicolon (;) as a separator.
Potential Code Changes:
File: pipenv/utils/dependencies.py
Function: dependency_as_pip_install_line
4. Code Snippet:
def dependency_as_pip_install_line(
# ... existing parameters
):
# ... existing code ...
elif vcs and vcs in dep: # VCS Requirements
# ... (existing logic for constructing git_req) ...
# Ensure VCS URL is preserved
if include_markers and dep.get("markers"):
line.append(f"{git_req}; {dep['markers']}") # append markers with semicolon
else:
line.append(git_req) # append the unmodified VCS URL string
# ... remaining code ...5. Additional Steps/Investigations:
Thorough Review of Requirement String Construction: Carefully review the dependency_as_pip_install_line function and related code to identify any other instances where VCS URLs might be incorrectly parsed or modified.
Extensive Testing: Develop test cases covering various VCS URLs with and without environment markers, ensuring the generated requirement strings are valid and the resolver handles them correctly.
Collaboration with Pip Maintainers: Given the interaction with Pip's stricter parsing, collaborating with Pip maintainers could help identify the root cause and ensure a robust solution.
Documentation: Update the Pipenv documentation to clarify the supported syntax for VCS dependencies in Pipfiles.
By addressing these points, Pipenv can ensure that VCS dependencies are correctly handled and prevent the ParserSyntaxError while accommodating newer Pip versions' stricter parsing requirements.
matteius
commented
4 days ago Fixed by: https://github.com/pypa/pipenv/pull/6276