action - Kick off a build when the base image updates

[castrojo]

We need a method in the github action to ensure we kick off a build when the Fedora base image changes so there's as little lag between builds.

Additionally it'd be neat to implement these in the other images in the org so when the base image builds it kicks off all the other ones.

I've looked at:

• twiddler/is-my-docker-parent-image-out-of-date@v1 (Trying this one in the experimental repo)
• https://github.com/lucacome/docker-image-update-checker - this won't work it only supports Docker hub and we need to support all major registries.

[castrojo]

@animeshn99 Mind researching this?

Right now Fedora is building the upstream images every 24h, so that's not hard to cron, but as the amount of images grows it gets annoying.

[animeshn99]

Sure thing.

[castrojo]

This looks like it might do what we need? https://github.com/GuiltyDoggy/ostree-container

[bsherman]

I have been investigating some options here. This is mostly a brain dump so I don't forget and maybe others find it useful.

Seems there's a couple issues that are conflated in my mind: 1) There's a use case for only building images when when the parent image has updated (or conditionally, like only 1 out of 24 times a day if there's a frequent timer for polling, and you want it to update each hour if the parent updated, otherwise always once per day)

This is something that could be addressed by a pattern similar to what Jorge shared:

This looks like it might do what we need? https://github.com/GuiltyDoggy/ostree-container

That solution could work, but it's not github action based, and even if it was, seems like it would need a PAT or other credentials.

Alternatively there's: https://github.com/marketplace/actions/is-my-docker-parent-image-out-of-date which is native to github actions and does not require a PAT, but does about the same thing.

2) The other thing is anytime a ublue-os/config build succeeds, it should trigger ublue-os/main build, and if that succeeds, it should trigger other dependent "in-org" repos to build (eg, nvidia and ubuntu/bluefin builds) and really... as the dependency tree gets more complicated, triggering gets more complicated.

As far as triggering goes, I think all the options I see for triggering non-current repos revolve around doing a remote "workflowDispatch". Example script: https://github.com/orgs/community/discussions/26323#discussioncomment-4217946 This does also require a PAT since it would be request to a repo other than current. It seems there are org-level PATs and secrets which could be used (not tested, this is based on reading some docs and looking at github settings), so at least, maybe, it could be stored as a single, re-usable secret.

The main concern I have about PATs is the 90-day expiration which I keep seeing people comment about.

3) this gets more complicated as we talk about other kinds of logic for determining when an image build should occur... like with akmods

@joshua-stone shared a good comment on this here: https://discord.com/channels/1072614816579063828/1083092181037891605/1083101580196319292

In summary, akmods don't need to build daily, or even when the parent image updates, probably only when the kernel updates.

[np22-jpg]

As discussed in Discord, there are a few options for monitoring quay.

1. Services such as newreleases monitor quay.io and are able to send a webhook that can trigger an action.
2. Compare timestamps inside a GitHub action, i.e. get the current timestamp from curl -s https://quay.io/api/v1/repository/fedora-ostree-desktops/base | jq -r '.tags."<VERSION_TAG>".last_modified' and use a compare it with a timestamp from a previous run. This approach does not rely on a third-party service, however, it's a little more messy. A cheap VPS + a webhook action might be better as to not cloud up the "Actions" tab.

As for querying for new packages in CI (for stuff like Nvidia), there's also a few options.

1. python3-dnf is available in Ubuntu's multiverse repo, though it'll be out of date. dnf can also be installed and used directly (with --repofrompath), too. This approach has the advantage of opening up overlay mounts as an option for reducing the amount of time it takes to collect metadata in CI, though I'm unsure if that could be useful for rpm-ostree.
2. Services like pkgs offer a (paid) API for getting the latest rpm versions.
3. For important things like nvidia drivers, release-monitoring can also be used for finding the latest package.

[bigpod98]

OK did some planning/thinking on how to do this in actions:

1. we need to have a separate repo that only does the checks and workflow triggering as we also need "storage" to store sha values of either images or git repos. Reason I wouldn't add these checks to repos like main is that they would likely just cause us to have to do a lot of PRs (due to current appropriate rules) and would just pollute commit log and changelog.
2. there are other things we need to watch out for as at the end of the day GHA isnt a statefull compute platform its a CICD pipeline
3. there are options to do outside of GHA but it would require compute time of one form or another and some sort of database. While compute could be lambda style serverless it would require some sort of state store like object storage or more properly some sort of database like KVP.
4. either way this wont be hard to implement as it is just pulling image or git repo and comparing hash value and running a curl to web api and updating value in data storage used
5. i havent looked into packages i just saw while reading this issue but that shouldn't be any harder i guess but we would need a list of software to check again stored in our storage solution used

[bigpod98]

i have been looking this this further, while yes in theory this can be done with actions it just shouldnt be done. it would bve best to use either some sort of service which currently to my knolwedge doesnt exist. So it would have to be custom.

This could be architected as proper service or more thrown together based on how well we want to make it

[ArtikusHG]

@bigpod98 Starting with F39 the images will be published onto quay.io. From the notification docs:

Webhook POST

An HTTP POST call will be made to the specified URL with the event’s data (see above for each event’s data format).

When the URL is HTTPS, the call will have an SSL client certificate set from Quay.io. Verification of this certificate will prove the call originated from Quay.io. Responses with status codes in the 2xx range are considered successful. Responses with any other status codes will be considered failures and result in a retry of the webhook notification.

It is possible to trigger a GitHub action with an http request.

I may be missing something, but this does seem like a properly architectured solution. If this won't work for some reason I'm missng, I apologize for the noise.

EDIT: the only problem I see is that it seems Fedora devs have to add our endpoint to their repository settings (?) but we could probably work around this.

[bigpod98]

We are not talking just about fedora images in theory we can use oci image as sources for binaries not available as rpm packages

[castrojo]

Yeah but we could do this for the main repo right? We can also ask Fedora for the endpoint, worse case they say no?

[ArtikusHG]

Yeah but that will probably cause a mess if we decide that we want to implement tracking of other OCI images (?)

I think we could do something along the lines of a GitHub action that checks every hour or so for changes in all the images, and if it detects a change, it triggers the build action. Not exactly real-time though....

So I guess the question is how important it is to support other OCI images but having to do this hacky check every hour vs just supporting the base image but in a clean easy way.

[bigpod98]

github action quickly falls out of question becuase how do you know image has changed aka you need to store a hash. (there are hacky way i thought of)

second option im honestly not that fan off becuase again it pigeon holes us into only having this for base fedora images for 1 and 2 if they would even be willing to give us that kind of automated ping(besides i think it needs to be authenticated call anyway so we would be giving them a pat which would every so often need to be changed anyway which is more work for them) 3. it would give us very little flexibility.

if we want to do this properly and clean i still think a lambda function(or any other event driven serverless function) and smallish state store this would facilitate addition of any other type of checking at package level or any other level specially if we ever expanded

[ArtikusHG]

how do you know image has changed

There are multiple ways. The simplest of them would be to just write the hash to a file in the repo. Yes, kind of hacky, but why not? We could also include the upstream image hash somewhere in the changelog/name/whatever of our downstream releases and pull it from there. GitHub actions has the benefit of being free of charge for public repos, just saying ;)

That being said, I really like the idea of a serverless function! I think we should dig in this direction.

[ArtikusHG]

Alternatively, we could store the latest built hash in the existing repo somewhere, and have the build action run more often (e.g. every hour), but do nothing in the case when every hash is the same.

[bigpod98]

Well while yes we could store hash in the repo itself im not a fan and i dismissed that from the start due to our policy for need of PR and approvals

[FruityWelsh]

Would a tool like renovate be helpful for your goals? Based on the discussion, a down side is it would require a token to made for the bot to run. It would create PR/MRs though and is a more git forge (GitHub, Gitea, Gitlab, etc) agnostic solution.

Unfortunately, RPM support PR seems like it was being added but was dropped, but may get picked back up by another contributor.