[Feature Request] Improved logging for failures

[logan-markewich]

Over at [llama-index](), we are huge users of pants, using it to test across 500+ python packages in a monorepo

Quite often, some package will depend on some external python package that updates and breaks things.

One thing I've noticed is that issues like envs failing to solve, or issues around interpreter constraints, are often very hard to debug.

When failing to solve venvs, usually I can read the dependencies and figure out which packages it might be failing on, but the actual name of the package or test its trying to build isn't reported in the error.

More recently, I hit some issues with interpreter constraints, and the error is even more vague

ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython==3.10.*,==3.11.* OR CPython==3.10.*,==3.12.* OR CPython==3.11.* OR CPython==3.11.*,==3.12.* OR CPython==3.11.*,==3.9.* OR CPython==3.12.* OR CPython==3.12.*,==3.9.*' failed with exit code 102.

stdout:

stderr:
Could not find a compatible interpreter.

Examined the following interpreters:
1.) /home/runner/.cache/pypoetry/virtualenvs/llama-index-MknZL8ZV-py3.10/bin/python CPython==3.10.15
2.)                                                             /usr/bin/python3.10 CPython==3.10.12

No interpreter compatible with the requested constraints was found:
  Version matches CPython==3.10.*,==3.11.* or CPython==3.10.*,==3.12.* or CPython==3.11.* or CPython==3.11.*,==3.12.* or CPython==3.11.*,==3.9.* or CPython==3.12.* or CPython==3.12.*,==3.9.*

Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

make: *** [Makefile:14: test] Error 1

While the error is clear enough, what isn't clear is where and what package/tests in my monorepo is causing the issue 😅

What I would like to see is just, when something goes wrong, pants doing its best to report what/where the issue is

[jsirois]

Well, I can't speak for Pants, but the underlying error message here comes from Pex and it does assume you know what comma means in the Python ecosystem (it means AND [^1]). Knowing that, here's the breakdown:

1. impossible: CPython==3.10.*,==3.11.*
2. impossible: CPython==3.10.*,==3.12.*
3. CPython==3.11.*
4. impossible: CPython==3.11.*,==3.12.*
5. impossible: CPython==3.11.*,==3.9.*
6. CPython==3.12.*
7. impossible: CPython==3.12.*,==3.9.*

So of the 7 OR'd interpreter constraints, 5 are impossible to ever satisfy. You can't have a Python interpreter that is both version 3.10 and version 3.11 (example item 1) at the same time. That's impossible! Maybe you meant to write CPython>=3.10,<3.12 for item 1? So you are left with 2 valid interpreter constraints in 3 and 6 leaving: CPython==3.11.* OR CPython==3.12.* and the interpreters identified in the error message are both 3.10 and so, of course, don't match.

Maybe, though, by "While the error is clear enough," you mean you understood this part of things? If so though, it's weird you include that example without correcting the 5 invalid ICs in the repo, which you could presumably find without more info by simply grepping pants.toml + BUILD files. For example, this: https://github.com/run-llama/llama_index/blob/044a439dc2fda53c991e49a43e4e8e652dd8a735/pants.toml#L22 Should probably be just:

[python]
interpreter_constraints = [">=3.9,<3.13"]

It means the same thing and avoids the Python ecosystem non-standard OR you get when you make a list of these things in Pants.

[^1]: Search for "The comma (“,”) is equivalent to a logical and operator" here: https://peps.python.org/pep-0440/#version-specifiers

[jsirois]

Yeah, Pants is probably doing incorrect algebra somewhere, it looks like you never actually write CPython==3.10.*,==3.11.* anywhere in the repo. You might want to correct all of these to eliminate OR (2 or more items in IC list, and have all use a 1-item IC list that leverage the standard Python AND (,) operator as discussed above. That might work around what appear to be Pants bugs synthesizing interpreter constraints:

:; git grep interpreter_constraint -- '**/BUILD' | grep '",'
llama-index-integrations/embeddings/llama-index-embeddings-mistralai/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/embeddings/llama-index-embeddings-vertex/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/evaluation/llama-index-evaluation-tonic-validate/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/extractors/llama-index-extractors-marvin/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/indices/llama-index-indices-managed-google/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/indices/llama-index-indices-managed-vertexai/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/llms/llama-index-llms-litellm/tests/BUILD:  interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/llms/llama-index-llms-llama-api/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/llms/llama-index-llms-vertex/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/program/llama-index-program-guidance/tests/BUILD:    interpreter_constraints = ["==3.9.*", "==3.10.*"],
llama-index-integrations/question_gen/llama-index-question-gen-guidance/tests/BUILD:    interpreter_constraints = ["==3.9.*", "==3.10.*"],
llama-index-integrations/readers/llama-index-readers-firebase-realtimedb/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/readers/llama-index-readers-firestore/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/readers/llama-index-readers-gcs/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/readers/llama-index-readers-lilac/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/readers/llama-index-readers-pandas-ai/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/readers/llama-index-readers-sec-filings/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/readers/llama-index-readers-whatsapp/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/response_synthesizers/llama-index-response-synthesizers-google/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/retrievers/llama-index-retrievers-vertexai-search/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/storage/docstore/llama-index-storage-docstore-firestore/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/storage/index_store/llama-index-storage-index-store-firestore/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/storage/kvstore/llama-index-storage-kvstore-firestore/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/tools/llama-index-tools-google/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/tools/llama-index-tools-text-to-image/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/vector_stores/llama-index-vector-stores-deeplake/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-integrations/vector_stores/llama-index-vector-stores-lancedb/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"]
llama-index-integrations/vector_stores/llama-index-vector-stores-vertexaivectorsearch/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-packs/llama-index-packs-deeplake-deepmemory-retriever/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-packs/llama-index-packs-deeplake-multimodal-retrieval/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-packs/llama-index-packs-gmail-openai-agent/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-packs/llama-index-packs-panel-chatbot/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-packs/llama-index-packs-ragatouille-retriever/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-packs/llama-index-packs-vanna/tests/BUILD:    interpreter_constraints=["==3.9.*", "==3.10.*"],
llama-index-packs/llama-index-packs-zenguard/tests/BUILD:  interpreter_constraints=["==3.9.*", "==3.10.*"],

[logan-markewich]

@jsirois thanks for the tip (and I didn't mean to come off as rude or anything btw, no hostility here 😁 )

Fumbling interpreter constraints aside, maybe a better example to illustrate https://github.com/run-llama/llama_index/actions/runs/10925018699/job/30325491519

This timed out trying to solve dependencies. Totally fair, I think some python version on something changed, causing this to timeout. But from there error, it's not indicated what test or package this was building for. I was able to guess based on the deps it was installing, but it's not always that easy.

I'm not sure about the underlying implementation details on how pants interacts with pex, just felt like when things go wrong at this stage, it might be possible to report what/where.

In any case, if that's not possible, feel free to close this

[jsirois]

For that error, you're seeing the direct output of Pip. There is really nothing more Pex or Pants could do for that one, except, as you allude to, maybe blame Pip. It was an explicit goal of Pex to kill it's bespoke resolver years ago in favor of just using Pip. Presumably the gold standard, but agreed in advance, the situation in reality is a mess.

You can ensure you're using the latest Pip (--pip-version latest when using Pex, not sure how you control this from Pants). Lots of example issues over in Pip about this, but here's one: https://github.com/pypa/pip/issues/12754

[logan-markewich]

Right, definitely an issue with pip.

I'm guessing there's no way for pants to report what test/package/path caused pip to break?

[jsirois]

Nope. If you read the issue I pointed to, even Pip folks aren't sure yet!

[jsirois]

I have no say on this issue, I maintain Pex, not Pants. But, if I were you I'd at least push hard on Pants killing interpreter constraint OR support. It's non-standard and confusing and, I think, handled buggily in cases like yours. Unlike Pex, Pants typically has no problems deprecating and removing features; so I'd think it could remove this foot gun.

[jsirois]

I've been doing a general sweep over Pex issues over the last few months and found https://github.com/pex-tool/pex/issues/432 which would have Pex fail fast when Pants handed it something like ==3.11.*,==3.12.* or else warn in the case you presented in the OP where 5 out of the 7 constraints were impossible but 2 were valid. Pex has since grown the technology to make that sort of determination; so I'll pick that up and get out a Pex release with that enhancement.

[jsirois]

@logan-markewich for the OP interpreter constraint message you'll now see the following when Pants upgrades to Pex 2.20.1 (or you do): https://github.com/pex-tool/pex/pull/2542/files#diff-5d77c74e67c3b985b12a434cde0c358f59c3e381f4cb033e86e7a2046048c018R57-R80 ... assuming Pants shows warning stderr output from tools it calls. Hopefully that's better.