search

JetBrains / intellij-platform-plugin-template

Template repository for creating plugins for IntelliJ Platform
Apache License 2.0
2.97k stars 587 forks source link

Fine-grained GitHub Actions for drastically faster CI #453

Open WarningImHack3r opened 3 months ago

WarningImHack3r commented 3 months ago

Describe the need of your request

Currently, the CI workflow is extremely slow (10-20 min average). It's not a big deal, but when you have to accumulate a push, a dependency update merge, another run for another dependency PR, another one for the second merge, and so on, it becomes very time-consuming and blocking for no real reason.

I say blocking because you have to sequentially wait for each one so that your release draft can actually be created, so you can finally publish it, wait for another PR workflow with the CHANGELOG.md update, and a final last one when you merge the changelog PR.

It's tedious, slow, and uses GH Actions bandwidth for practically nothing. I can sometimes waste 30-60min for a 5-line fix that I release alongside 2 Renovate PRs.

Proposed solution

The solution lies in 2 words: conditional execution.

I've used this technique in some of my recent projects, where I make my CI smart: it only runs the steps it needs to run depending on the changes I make.

What would I change for our case? 2 things:

How would I apply them?

What would those changes mean?

Taking our same example above, we'd go from:

To:

Would that be a big deal?

With rough average estimations for a relatively medium codebase of:

And being generous because Post Setup Gradle can be extremely slow, sometimes doubling the job's duration but that's not on you, we would go from ~72 min (8 9 min) to ~19 min (1 9 min + 5 2 min + 2 0s) of run for a casual workflow with hotfix releases and dependency management.

Additional actions improvements

If you want to go beyond, maybe improving the run duration of your own actions, especially ./gradlew runPluginVerifier and JetBrains/qodana-action, which are the slowest of all (maybe parallelizing stuff like IDE checks), it'd be amazing!

About the Release workflow, it would also be great not to have it fail if the CHANGELOG.md is already up-to-date!

If you guys also have an idea of why and how Post Setup Gradle can take up to 4-5 min, I'd love to hear from you!

Additionally, I'd love to see Qodana reports in PRs not getting sent on closed/merged PRs.

Conclusion

Please consider this change that would have a massive cumulated impact on all the people using this template, both in time for plugin developers and in computation for our friends @ GitHub

Alternatives you've considered

Waiting or going back and forth on GitHub without being focused long enough on another task or even canceling some workflows

Additional context

No response

YannCebron commented 1 month ago

Thanks for your detailed suggestions, we are very much interested in optimizing the build/CI performance for plugin projects.

The suggested optimizations to skip execution of Gradle tasks make a number of assumptions that may or may not be true in certain plugin projects (e.g., fixed folder names, assuming source directory folder names, etc.). While it certainly can be done and documented, it seems more suitable to try and optimize the GH actions workflow itself without relying on any "hardcoded" rules that all projects started from this template must obey to work.

We'll look into the other points and welcome any further suggestions.

ChrisCarini commented 2 weeks ago

Hi @WarningImHack3r -

I would love to see this, as well! Thanks for enumerating everything in so much detail!

  • I would skip releaseDraft if a release with the same name has already been released

I believe I accomplished similar in my plugins; one example here: https://github.com/ChrisCarini/sample-intellij-plugin/pull/267 - essentially, I only wanted a GitHub release to be created if the version of the plugin that was currently being built (in the CI run) was different than the most recently released version ( on GitHub).

  • I would skip <gradle task> when \

I suspect as Yann said above, all IJ platform plugins using GHA might not want to have the same set of 'hard-coded' rules (that's to say, different plugin authors/maintainers may have their own workflows (not GHA workflows, but development/release workflows) that they want to accommodate).

However...

☁️ High-level

For myself, I want most all changes (with some exceptions, see below for details) to result in a new version of my plugin(s) being released, and I want to have and (a) fast CI and (b) minimal 'touch points' after I merge in 'the significant' change (e.g. if a new version of IJ is released, I want to have one PR with the necessary changes, and once that CI passes, I want the rest to be (a) fast, and (b) 'touch-less').

Because of what I describe above, I believe I accomplished something similar for my plugins; example below (its spread across 2 commits):

🔩 The Details

My 'conditions' are more trivial than the ones you enumerate here, but they've been working well for me and my workflows (GitHub workflows, and personal ones for features & dependency changes like new IDE version, gradle, or GHAs).

It's been a while since I made these changes, but my thought process was something along the lines of:

  1. There are several types of changes my IJ plugins have:
    1. code changes (features, bug fixes, etc)
    2. dependency upgrades (gradlew, java deps, IDE version, GitHub Actions, etc)
    3. 'metadata' (README.md, CHANGELOG.md, etc)
  2. I really only want CI (the build & testing aspects) to run for any 'meaningful' changes to code - that encompasses 1.i & 1.ii for me. I don't really care to have CI run for 1.iii, since those changes are usually insignificant and there should be a zero-chance they'd affect anything in the actual plugin/code/outcome of CI from the prior change. More specifically, 1.i will certainly result in a different artifact (because, some code is changing), and 1.ii could likely to result in some difference in what's being run in CI as well (some change to gradle, some change to java dep, IDE SDK APIs changed, GHA changed behavior in some way, etc).
  3. I don't want my CI to run for CHANGELOG.md changes that result from a newly published GitHub release (and, thus, a new plugin version).

🧑‍🏭 Example

As an example of this (the big, and annoying, overhead for me was CHANGELOG.md PRs & post-PR-merge CI), below is the timings for a recent-ish CI:

Screenshot 2024-07-02 at 07 39 24

🙏 I'd love feedback & ideas, if you have any!

I'd love to know your thoughts, and hear if you have any specific suggestions for my own GHA workflows! I maintain several IJ plugins, and try to keep their 'scaffolding' the same across (I have a central repo I push changes to things like gradle build files, or GHA workflows) for my own maintenance ease.

WarningImHack3r commented 2 weeks ago

I would love to see this, as well! Thanks for enumerating everything in so much detail!

It impacts me so much that I had to provide as much detail as possible to increase my chances of getting actual changes made :( Additionally, having some exp on optimizing workflows, I know some stuff is doable.

I suspect as Yann said above, all IJ platform plugins using GHA might not want to have the same set of 'hard-coded' rules (that's to say, different plugin authors/maintainers may have their own workflows (not GHA workflows, but development/release workflows) that they want to accommodate).

To be fair, as my "rules" rely on basic and existing Java apps architecture, I don't think I'm making crazy assumptions... As there is a big room for improvements, I'd prefer these people not following the standards to adapt their workflows themselves as a consequence, and specify they'd have to do so in the README

For myself, I want most all changes (with some exceptions, see below for details) to result in a new version of my plugin(s) being released, and I want to have and (a) fast CI and (b) minimal 'touch points' after I merge in 'the significant' change (e.g. if a new version of IJ is released, I want to have one PR with the necessary changes, and once that CI passes, I want the rest to be (a) fast, and (b) 'touch-less').

I'm not sure to get what you're describing here

It's been a while since I made these changes, but my thought process was something along the lines of:

Yep! That's basically the minimal "smart" workflow I'd like to have, and that I describe in my original message; it's easy to filter in/out some file names, agnostic from any file architecture you want and that would bring high QOL for such common and annoying use cases as changelog PRs!

🙏 I'd love feedback & ideas, if you have any!

I don't have much time right now and I'm not sure to get what you described in your "High-level" section, but AFAIK we pretty much have the same needs and we could benefit from the same improvements I describe here and would like!

ChrisCarini commented 2 weeks ago

For myself, I want most all changes (with some exceptions, see below for details) to result in a new version of my plugin(s) being released, and I want to have and (a) fast CI and (b) minimal 'touch points' after I merge in 'the significant' change (e.g. if a new version of IJ is released, I want to have one PR with the necessary changes, and once that CI passes, I want the rest to be (a) fast, and (b) 'touch-less').

I'm not sure to get what you're describing here

TL;DR: I want the process I personally use to release plugins to be (a) fast, and (b) have minimal human involvement from myself. I have 11 plugins I maintain and try to release for each new version of IJ that JB releases, and time is precious - these drive my desire for (a) and (b).

Yep! That's basically the minimal "smart" workflow I'd like to have, and that I describe in my original message

Awesome! Glad we have similar desires/goals here! Feel free to look at my workflow modifications as a reference for your own. 😄

I don't have much time right now and ... AFAIK we pretty much have the same needs and we could benefit from the same improvements I describe here and would like!

Yeah, we're all limited on time. Great to hear it sounds like we both have similar needs and could benefit from each other's experiences/discussions. Like I mentioned above, feel free to use the modifications to the workflows in the repo (one of many I have) linked in my previous message as a starting point - if you make modifications for yourself, I'd love to know about them to see if they are modifications I'd also be interested in (as a joint benefit/learning, if you will). Perhaps between the both of us (and possibly others in the community), we can then use this first-hand experience to suggest actual changes in the form of PR(s) and see if the JB team is interested in accepting them for all!

WarningImHack3r commented 2 weeks ago

TL;DR: I want the process I personally use to release plugins to be (a) fast, and (b) have minimal human involvement from myself. I have 11 plugins I maintain and try to release for each new version of IJ that JB releases, and time is precious - these drive my desire for (a) and (b).

11 plugins is indeed a lot, but avoiding waste of time is always a plus no matter how many repos you have anyway!

Awesome! Glad we have similar desires/goals here! Feel free to look at my workflow modifications as a reference for your own. 😄

TBF I'm not really willing to do it myself, both for time reasons, laziness and also cause it the JB teams happens to push improvements I don't want to have to rollback my changes...
But yeah I may take a look at what you did if I'm too pissed off someday!

Perhaps between the both of us (and possibly others in the community), we can then use this first-hand experience to suggest actual changes in the form of PR(s) and see if the JB team is interested in accepting them for all!

That indeed looks like a great idea! I'll keep you updated ;) thanks for you kind words, research and POC!!