Improve CI Build time - Githubissues

eto-ai / rikai

Parquet-based ML data format optimized for working with unstructured data

https://rikai.readthedocs.io/en/latest/

Apache License 2.0

136 stars 19 forks source link

Improve CI Build time #621

Open eddyxu opened 2 years ago

eddyxu commented 2 years ago

Current GHA takes about 15 minutes to finish all tests. For rikai-scale project, this is not necessary.

Let's do some research to see where can we accelerated. A few potential low hanging fruit.

Reduce the total number of build matrix.
- Only test jdk8 and scala 12 against all Python version.
- Only test jdk 11 + scala 13 against the latest supported python version.
- Prebuild mlflow and spark docker images to speed up Python CI task.

da-liii commented 2 years ago

For pretrained torchvision models, the pt files are saved $HOME/.cache/torch/hub/checkpoints. Just cache it in github action would save a lot networking time.