Open tiran opened 3 weeks ago
I agree this needs to be part of a build pipeline. Does it need to be a core part of fromager?
If we look at fromager as producing a "set of wheels" as output, then we might want to test them together as well as separately. Are you thinking fromager would test individual wheels as they are built, and fail the build if a test did not pass? Then at the end of the job something would test the collection together?
As a maintainer, I would like to have a system to run gating tests and sanity checks for a wheel that have been built by Fromager. The tests should be executed after a wheel and all its dependencies have been built and before the source directory of the package has been removed. Fromager should automate the process of creating a test virtual env, installing the package, and running tests.
Proposals
overrides/gating/{override_package_name}/test*.[py|sh]
(e.goverrides/gating/flash_attn/test_something.py
oroverrides/gating/torch/test.sh
).os.environ
to each test fileFROMAGER_VARIANT
,FROMAGER_SDIST_ROOT
, and whatever else we find useful. This lets test scripts locate the unpacked sources or change different aspects for each variant.overrides/gating/{override_package_name}
as their CWD.python -m test_$name
Possible enhancements for later