Closed sutaakar closed 4 months ago
@astefanutti Ready for second round of review. Right now GH action runs both CPU and GPU tests. Thinking whether we should run only GPU tests on PR check, leaving CPU just for possible local execution (will speed up PR check). Dropped MNIST fashion example, all tests are using "plain" MNIST, to keep changes smaller. Can be ported to some other example later if needed.
[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: astefanutti
The full list of commands accepted by this bot can be found here.
The pull request process is described here
Issue link
RHOAIENG-9004
What changes have been made
Adjust GitHub e2e workflow to setup and install NVidia operator. Convert Ray tests to use MNIST fashion test case on GPU. Adjust
TestMNISTPyTorchAppWrapper
to use GPU. As a workaround for having just one GPU the tests are executed sequentially.I also had to alter uploading CFO image to be loaded by KinD CLI. The reason is that GPU image uses Docker and doesn't interact well with locally created registry.
Edit: Thinking whether we should add GPU tests next to existing nonGPU ones, or rather replace them. Having both will increase number of tests, on the other side non GPU tests are faster and can be ran anywhere.
Verification steps
Run e2e PR check - GPU is made available on KinD cluster and tests use it.
Checks