Verify pixi schema against CLI

[bollwyvl]

Problem description

references

• 2055

elevator pitch

Check for rust- (and cargo-) sensitive changes vs hard-coded values in the generated schema.

motivation

New platforms have appeared that the schema didn't know about.

Changes like this are pretty much guaranteed to happen again, and other classes of issues can be caught per-keystroke.

design ideas

During pixi r test-schema, for each schema/test/invalid/{name}.toml, e.g. bad_feature_platform.toml:

• look for files called {name}.{action}.stderr.txt
• if any exist, in a temporary folder
  • copy to the manifest to pixi.toml
  • run $PIXI_BIN {action}
  • compare the output to the expected

alternatives

• consume the schema.json in the rust-based test suite
  • in the case of platforms, there's probably a list somewhere
  • this would make it harder to contribute structural schema changes (such as #2055)

[ruben-arts]

I 100% agree with this. Most of these hard coded things are not moved to often so we could just make simple code parser and verify that it's the same in python and in Rust. Otherwise making Python bindings for these parts and using the bindings in the model.py could also be helpful.

[bollwyvl]

Welp, could PR something that was like:

flowchart LR
  build([pixi r build]) --> pixi.exe -- $PIXI_TEST_BIN --> test-schema([pixi r test-schema])
  schema([pixi r generate-schema]) --> schema.json --> test-schema

This would be outside of my wheelhouse, but probably more efficient:

flowchart LR
  test-schema([pixi r test-schema])
  schema.json --> test([pixi r test])
  schema([pixi r generate-schema]) --> schema.json --> test-schema