Use pip-resolver-benchmarks and Pip code to create "real world" resolvelib tests?

[notatallshaw]

I have an idea to use pip-resolver-benchmarks and Pip code to create tests that could exactly model real world scenarios to test resolvelib.

In particular I would want to add test cases from https://github.com/sarugaku/resolvelib/issues/134 where they are represented by a json, then as part of the test prep get the pip-resolver-benchmarks code to build the wheels, and then have enough Pip code for this to fail in the current resolvelib.

While I would try and keep it minimal as possible it probably would be quite a lot of code from both pip-resolver-benchmarks and Pip to get this working. So I understand if it's not acceptable.

In particular @pradyunsg do you have any thoughts or objections?

[uranusjr]

Sounds like a good idea to me. The suite already contains some real-world cases I stole from other resolvers before pip started using resolvelib, and now pip has its own real-world cases, it makes sense to add them here.

[notatallshaw]

I've pivoted a little on this, and am now focusing on packse tests, once I have packse tests working for pip (https://github.com/pypa/pip/pull/12580) I am going to see if I can use the pip-resolver-benchmarks collection tool to make jsons that I can transform into packse jsons that could be used for packse to make "real world" tests for either pip or resolvelib or both. But I may pivot again, depending how packse and pip-resolver-benchmarks update over time.

[notatallshaw]

Upon writing my first functional test case in https://github.com/sarugaku/resolvelib/pull/152 I'm now of the opinion the easiest approach would be to write some script(s) that turns the pip-resolver-benchmarks format into something that the PythonInputProvider can read.

It would also be useful to have an easy way to minify the JSON so it only included just enough to trigger the errors.

For now I will close this issue.