search

overhangio / tutor-mfe

This plugin makes it possible to easily add micro frontend (MFE) applications on top of an Open edX platform that runs with Tutor.
GNU Affero General Public License v3.0
22 stars 95 forks source link

feat: invalidate build cache on upstream changes #131

Closed arbrandes closed 1 year ago

arbrandes commented 1 year ago

This will invalidate the image build cache for each MFE whenever:

  1. A direct URL to the repository's branch refs is provided

and

  1. The head of the branch being built changed between builds.

For Github, this is available via [1]. For Gitlab, via [2].

[1] https://docs.github.com/en/rest/git/refs?apiVersion=2022-11-28#get-a-reference [2] https://docs.gitlab.com/ee/api/branches.html#get-single-repository-branch

arbrandes commented 1 year ago

Just tested this, and BuildKit seems to be smart enough to only rebuild the MFE in question. (And yes, only when there's an upstream change.)

arbrandes commented 1 year ago

@ghassanmas,

this is expected to be useful when working with branchs not tags, Am I right?

Correct! This only makes sense when rebuilding the tutor-mfe image from branches. I figure it will be most useful on the couple of months between the cutting of master for the next release and the release itself, when we're frequently rebuilding from the release master branch.

But it could also be useful for nightly rebuilds, or for people that have custom branches that get rebuilt often.

Also, while I included it in the patch, there's really no reason to have default values that point to tags in the master branch. It was more a way to demonstrate syntax; we can remove those if people think this is a good idea.

arbrandes commented 1 year ago

Re-tested, works fine. Merging!

regisb commented 1 year ago

Hmmmm but this is going to cause the build to fail in nightly, right? When I reviewed the PR I did not realize that the ref url would be different for a branch.

arbrandes commented 1 year ago

@regisb, This is what I did for nightly: https://github.com/overhangio/tutor-mfe/commit/e3326d953f84832c4a62e36a0e021caac47ffff3. (I pushed directly so as not to break the build.)

It's going to have to be done whenever going from tags to branches and vice-versa. Unfortunately, it's just the way the GitHub API works. :/

regisb commented 1 year ago

This situation is going to be difficult to handle in the Quince upgrade. I'd rather avoid commits that are exclusive to the nightly branch. Instead, I suggest that we switch dynamically from "tags" to "heads" based on the value of tutormfe.__about__.__version_suffix__.

Oh, and I expect that some users will also face issues when they manually set OPENEDX_COMMON_VERSION to a branch...

arbrandes commented 1 year ago

Ah, didn't know about __version_suffix__! Let me give it a shot.

But changing OPENEDX_COMMON_VERSION is going to be an issue, indeed. I wonder (again) if we should just leave default refs out of master. That means nightly will have an additional commit to optimize rebuilds, but... it does already have a couple of nightly-specific commits anyway. 🤷🏼‍♂️

regisb commented 1 year ago

it does already have a couple of nightly-specific commits anyway.

Does it? We should really try to avoid this situation as much as possible. That being said, I acknowledge that the design of the nightly/master branching model leaves much to be desired. I'd be happy to hear other ideas.

arbrandes commented 1 year ago

Ok: tutor has a couple of nightly-specific commits, not tutor-mfe. I'm with you, though. We should avoid this as much as possible.

As for master vs nightly, the situation is always going to be a bit tricky. I'm more used to the master-is-unstable paradigm, so that it is the release branch that has branch-specific commits. But with publishing versioned packages, that wouldn't work very well.

I know of one alternative we can consider. If nightly were an integration branch, consisting of an octopus merge of several feature branches, then picking which ones to pull into master should be easy later.

For instance, we could have a series of long-lived branches based on master, such as:

Then, whenever desired, the nightly branch is recreated via:

git checkout nightly
git reset --hard master
git merge bug-fix-for-master new-feature nightly-base

The next release branch could be managed as an integration branch, too. The advantage is that there is always a single source of truth for a particular changeset, and it gets pulled into whichever integration branch requires it. That way we always know what shouldn't eventually go into master, but at the same time we can build a single artifact for testing.

The disadvantage is that integration branches don't (re)create themselves, and have no memory. We'd want a script to do the integration process, and it would need to be modified whenever a new branch needs to be included.