devendragovil
commented
1 year ago @ninamiolane
Analysis
I have analyzed the runtime for all unit tests. Hypergraph Tutorials indeed do take the longest durations. Please find the times of the longest 5 tests here:
| Category | Name | Run Time (sec) |
|---|---|---|
| Hypergraph | DHGCN. | 208 |
| Hypergraph | Hypersage | 176 |
| Hypergraph | UniGCNII. | 81 |
| Hypergraph | UniGCN | 42 |
| Simplicial | Scone | 27 |
My observations:
- Individual test times are not that outrageous.
- It takes really long because all the tests are running sequentially
Deep Dive (DHGCN Tutorial)
All steps are taking reasonable amount of time (< 5 secs) except the last step which is a 5 epoch training run for the DHGCN Hypergraph TNN.
Observations
- Individual train times do seem reasonable to me (please correct me if I am wrong). These might speed up with GPU access
- Environment is built repeatedly. For tutorials the libraries are imported repeatedly.
- All tests (until recently) were being run sequentially.
Recommendations/Solutions
- We can arrange for GPU for the test-suite. I don't think Github actions provides a runner with GPU, we will need to arrange our own runner, which can be configured. However, configuration might be time consuming and hosting a GPU instance might be costly.
- We can do aggressive caching for our environment as well as libraries being imported.
- We can run tests in parallel. There are many libraries like pytest-xdist and pytest-split that enable this. Since Github Actions runners are single core, we can use the matrix strategy for parallelization. The tests can also be split based on the time they take to enable 5-7 (or as required) equally timed partitions. Since the longest test takes just over 3 minutes, that is the shortest time parallelization can achieve without making changes in tutorials themselves.
- A Naive Solution: Reducing number of epochs in tutorials. Reducing number of epochs in DHGCN from 5 to 1 reduces the time by a fifth, and DHGCN tutorial concludes within a minute.
ninamiolane
What?
Testing the tutorials on hypergraphs takes ~15 minutes, whereas testing the tutorials on other domains takes ~2-5 minutes (see screenshot).
There is probably one tutorial on hypergraphs that takes very long and slows down the whole github action workflow.
Find out which one and whether it can be accelerated.
Why?
A slow testing workflow slows down all the contributors, who have to wait for all tests to pass before being able to move on.