Research spike: Run CI integration tests on GPU runners

[deprit]

Given the differences in Armory-Library dependencies installed on CPU versus GPU instances, we would like to make our CI tests more comprehensive by evaluating them on GPU-enabled runners. The research spike should answer the questions below.

1. What subset and on what actions should CI tests run on GPU instances? Should we create a separate workflow to reduce the frequency (and cost)?
2. Is it possible and how expensive would it be to use Github Actions GPU runners?
3. Should we host our own GPU runners on noether to reduce cost? How difficult and how much maintenance does this require?

[deprit]

As pointed out by @treubig26 and @jay-soni-26, we should reject option 3, self-hosted runners, as a security risk for a public repository.

[deprit]

From @jay-soni-26, cost estimate below for option 2, Github GPU runners.

Our builds take about 20 minutes. With the same amount of time on a Linux 4-core GPU that would be $1.40/build at $0.07/minute (https://docs.github.com/en/billing/managing-billing-for-your-products/managing-billing-for-github-actions/about-billing-for-github-actions#per[…]tes). We currently seem to do an automated daily build and then also builds for PRs. If we did 60 builds a month, that would be $84.00. It's not clear how storage cost is calculated but we don't seem to use much so I'd assume not much. If we started out with $100, I'm pretty sure that would be more than enough for a month. A couple optimizations to reduce cost: not doing the automated daily builds on the GPU and/or reducing frequency of the automated builds.

[deprit]

We are able to provision a hosted large (GPU) runner for use by Armory-Library.

[deprit]

As all questions have been answered, we will implement a GPU workflow using the ubuntu-gpu hosted runner.