CI: Define reusable workflows and composite actions

[akihironitta]

Proposed refactor

Some of the steps/workflows are repetitive across workflows/repositories, which makes it very time-consuming to make sure that all of our workflow files are updated in the same way.

Items to be decoupled

• caching-related steps
• downloading legacy checkpoints
• dependency installation
• ... (open for suggestions!)

Motivation

To improve maintainability.

(I think we should first focus only on this repository, but it will also beneficial for other repositories in the future as these decoupled actions/workflows can be called from other repositories.)

Pitch

1. Decouple repetitive steps from existing workflows, and pack them into composite actions in the following directory:

   .github/actions/*/config.yml

2. Decouple repetitive workflows from our repositories and define reusable workflows (reusable workflows are different from composite actions above).

   .github/workflows/_*.yml

Additional context

• Planning to utilise:
  • reusable workflows: https://docs.github.com/en/actions/using-workflows/reusing-workflows#reusable-workflows-and-starter-workflows
  • custom actions: https://docs.github.com/en/actions/creating-actions
• Similar to torchmetrics CI: https://github.com/PyTorchLightning/metrics/tree/master/.github/actions

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Lite: enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

• Bolts: Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

• Lightning Transformers: Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

cc @carmocca @akihironitta @borda @justusschock @awaelchli @rohitgr7

[Borda]

you can check PoC from Metrics: https://github.com/PyTorchLightning/metrics/tree/master/.github/actions/caching

[Borda]

I belive that we have done this :otter: great work @akihironitta and @carmocca!