Closed notatallshaw closed 2 months ago
notatallshaw
commented
2 months ago I am giving up on using packse for now, I've created a sufficently minimal scenario with pip-resolver-brenchmarks. I will return to try and recreate the scenario with packse once I understand it better.
zanieb
commented
2 months ago Hey just want to acknowledge this, I'm really swamped but it's on my backlog and I'll try to take a deeper look at some point.
It looks like pip complains that pystac-client==0.3.2 can't be used in the benchmark scenario but in the packse one pip installs pystac-client==0.3.3 (which is allowed by the top-level constraints in both cases). It might be helpful to try uv on the benchmark scenario since it'll show a proof of why the dependencies can't be used, I'm guessing that's challenging though?
I'm also curious if allow_sdists_for is relevant. If you remove that does the benchmark do something different? Do we need to disable source distributions for the packages not in that list?
Did you try looking at the metadata of one of the package distributions built by packse? I see some syntax in your generated file like "requires_python": "!=3.0.*" and "asv~=0.6.0" which I'm not sure are supported by packse.
zanieb
commented
2 months ago Oh tragic I commented then my tab refreshed and I saw your comment closing this.
Thanks for following up!
notatallshaw
commented
2 months ago Hey just want to acknowledge this, I'm really swamped but it's on my backlog and I'll try to take a deeper look at some point.
Hey, thanks for acknowleding, I appreciate how swamped you are which is why I decided to close this, it was in service of what ended up being this PR where you can see the test I ended up creating: https://github.com/sarugaku/resolvelib/pull/152
I'm also curious if allow_sdists_for is relevant. If you remove that does the benchmark do something different? Do we need to disable source distributions for the packages not in that list?
I was just throwing stuff at the wall to see what or what wouldn't work.
Did you try looking at the metadata of one of the package distributions built by packse? I see some syntax in your generated file like "requires_python": "!=3.0.*" and "asv~=0.6.0" which I'm not sure are supported by packse.
The test case I have in that PR ended up not requiring requires_python, it does use ~= but that's incidental to the test itself. When I get a chance I will try changing the ~= in my packse scenario and see if that works.
I'm working on trying to get a process where you can put in real world requirements into
pip-resolver-benchmarkand get apacksescenario.I am motivated in particular by the requirements
"pandas<=1.4.0,>=1.3.5" "pystac<=1.8.3,>=1.8.2" "pystac-client<=0.3.3,>=0.3.2" "sat-stac<=0.1.1", which should install and does with uv but doesn't with pip.This is the the
pip-resolver-benchmarkoutput: https://gist.github.com/notatallshaw/de066471b0a5458b58e0204842f2e110And running tests it reproduces the bug in pip:
I have wrote a quick hacky script to try and convert the output to
packseformat:hacky-prb-packse-converter.py
```python import json def convert_to_packse(pip_resolver_benchmark_json): packse_json = { "name": "pip-resolvelib-bug", "description": "A pip/resolvelib bug (https://github.com/sarugaku/resolvelib/issues/134 / https://github.com/pypa/pip/issues/12317)", "root": {"requires": []}, "packages": {}, "expected": { "satisfiable": True, }, } # Load the input JSON with open(pip_resolver_benchmark_json, 'r') as f: pip_resolver_data = json.load(f) # Extract the root package information packse_json["root"]["requires"] = pip_resolver_data["input"]["requirements"] packse_packages = packse_json["packages"] for package_name, package_versions in pip_resolver_data["packages"].items(): packse_packages[package_name] = {"versions": {}} packse_versions = packse_packages[package_name]["versions"] for package_version, package_details in package_versions.items(): packse_versions[package_version] = {} packse_version = packse_versions[package_version] package_dependencies = package_details["depends_by_extra"] if package_dependencies: for extra_name, extra_dependencies in package_dependencies.items(): if extra_name == "": packse_version["requires"] = extra_dependencies continue if "extras" not in packse_version: packse_version["extras"] = {} packse_version["extras"][extra_name] = extra_dependencies if package_details["requires_python"] != "None": packse_version["requires_python"] = package_details["requires_python"] return packse_json # Example usage pip_resolver_benchmark_json = 'pip-resolver-benchmark-scenario.json' packse_json = convert_to_packse(pip_resolver_benchmark_json) # Write the output to a file with open('packse-format.json', 'w') as f: json.dump(packse_json, f, indent=4) ```And this is the output: https://gist.github.com/notatallshaw/723f19a55102413cf1fef66631498e08
Which packse successfully builds and serves, but when I run the requirements against the index server the installation works:
Please don't spend much time on this, but there's something obvious I'm missing please let me know.
P.S I have a fork of I have a fork of
pip-resolver-benchmarkto get this working, happy to publish the whole workflow if I can proove it actually works.