erlef / setup-beam

Set up your BEAM-based GitHub Actions workflow (Erlang, Elixir, Gleam, ...)
MIT License
373 stars 50 forks source link

Consider adding support for caching deps and build artifacts #293

Open starbelly opened 2 months ago

starbelly commented 2 months ago

I believe we discussed this before and did not think it was appropriate, mainly because cache invalidation is the second hardest problem in computer science 😉

Setting up cache in a project on your own is a trivial matter, though people disdain the redundancy of copy / pasta on every project, which I can understand.

setup-beam would either have to leverage a composite action, which doesn't sound great to me, or utilize github-toolkit (as we currently depend on today).

I think if this was added, it should be enabled by default. We'd also need to provide a way to bust the cache (i.e., a cache_version input). Github provides a way to delete cache in the UI, but it also annoying to go through IMHO.

The main caveats are here is when the cache invalid (e.g., an upload to github went weird), thus the need for a way to bust the cache.

Here is a very old example of a composite action, while I don't think a composite action is the way to go here, it still serves as some kind of example.

https://gist.github.com/starbelly/f423c2a3e059fa9c5e2647e4b5a7b175

paulo-ferraz-oliveira commented 2 months ago

You mean cache e.g. _build and stuff generated in the consumer's pipeline (like Dialyzer cache?)? What if they're not using rebar3, what assumptions are we to make? I maybe understood this wrong, but:

  1. what issue are you trying to solve?
  2. what interface do you propose to solve it?
starbelly commented 2 months ago
  1. The issue that others want to solve is repetetive copy/pasta (i.e., copy a cache solution from one project to another, get it wrong, edit, push, repeat until right).
  2. I'm not proposing an interface myself yet, this has to be thought about more is there are hidden complexities when it comes to caching, especially across different tools (rebar3, hex, etc). and repository structures (i.e., lib, single app, and umbrella). Rather, this is about a simple discussion of whether we'd like to start to think about those problems.
paulo-ferraz-oliveira commented 2 months ago

There's a lot to unpack here, for sure:

  1. what to we want to cache
  2. when to we want to cache
  3. how much control over the cache we want to give over configuration options
  4. ...
rhcarvalho commented 2 months ago

As reference points, these two "language/ecosystem" actions have built-in cache integration:

I don't think caching here needs to cover all possible use cases, as long as users can opt-out and go use actions/cache or something else when desired.

Here's a current cache usage I have in a workflow:

      - uses: actions/cache@v4
        with:
          path: |
            deps
            _build
          key: ${{ runner.os }}-elixir-${{ hashFiles('**/mix.lock') }}
          restore-keys: |
            ${{ runner.os }}-elixir-
paulo-ferraz-oliveira commented 2 months ago

Yeah, but then this turns into a bigger problem: we already support Elixir, Erlang and Gleam and intend on eventually supporting LFE (possibly Luerl). So now you have to maintain 4 cache "types", plus "when to bust them?" (?)

Looking at your example:

  1. you might be missing, for example, the Elixir version, since that should bust the cache
  2. the same goes for the Erlang/OTP version
  3. I've had many issues with restore-keys in the past, especially when stuff moves in the main branch and provokes incompatibilities, etc.

There's a host of issues the action will not solve (and would get unnecessarily complicated for) that we could probably curate in a community -driven Gist of good practices, or similar.

Then again, this could also be a different action that'd assume setup-beam exists, and you'd have something like

- uses: erlef/setup-beam-cache
  with:
    for: elixir

and then inside the specific Elixir rules would be maintained.

Or even several actions, if the differences are too big.

I just don't think, at this moment, this is a specialisation the action needs.