Consider sending fedmsg when stream metadata changes

[bgilbert]

Something might want to process stream metadata updates without polling.

[lucab]

Even in the case of polling, a "stream metadata updated" event can be used as a trigger to do an immediate refresh instead of waiting for the whole polling period. Assuming the event stream can be watched by the general audience (i.e. it does not contain sensitive data that requires ACLs), this is interesting for the Cincinnati backend.

[jlebon]

We should be able to do this pretty easily now that we're set up to send fedmsgs. See possible message template for this at https://github.com/coreos/fedora-coreos-tracker/issues/198#issuecomment-513944390.

[jlebon]

So releases today actually happen in three steps:

• the release job makes some things public and updates the release index
• a human updates the stream metadata in the git repo (e.g. https://github.com/coreos/fedora-coreos-streams/pull/167)
• the sync job syncs the stream metadata to S3

Technically, it's released as part of the first step, but it doesn't actually show up on the website until the last step.

I think we should just have a fedmsg for each. The first one could be e.g. org.fedoraproject.prod.coreos.release and the second one org.fedoraproject.prod.coreos.metadata-update. We'll need to add change detection to the sync job so we just no-op if nothing actually changed.

[jlebon]

Some context on this from IRC: this came up in a discussion with @AdamWill who was interested in these messages to be able to hook FCOS releases into openQA testing. One suggestion I made was to hook it to builds instead of releases so tests happen earlier. But we also today don't emit messages on new builds. So in total I think there would be 4 useful messages to add:

• build started
• build stopped (with final result)
• build was released
• stream metadata was synced

For the first two, see the first bikeshed message payload at https://github.com/coreos/fedora-coreos-tracker/issues/198#issuecomment-513944390.

(I realize this is hijacking the original ticket a bit, though it's good to think about how all these messages fit together.)

[AdamWill]

for my purposes, the key requirements are that the message be emitted when a new set of artifacts (specifically the "live" ISO and/or the disk images, assuming those can be booted if attached to a VM and fed an ignition file somehow, I haven't tested that yet) is available for download, and that a consumer can discover or reliably infer (by some kind of predictable and stable pattern) the location of the artifacts from the contents of the message.

[dustymabe]

So in total I think there would be 4 useful messages to add:

• build started
• build stopped (with final result)
• build was released
• stream metadata was synced

SGTM

[jlebon]

@AdamWill this is in production now. Here is an example fedmsg emitted when a build is done:

https://apps.fedoraproject.org/datagrepper/id?id=2020-81657c3b-9703-431a-8c3c-0b409743fac4&is_raw=true&size=extra-large

Copying relevant snippet for posterity:

{
  "topic": "org.fedoraproject.prod.coreos.build.state.change", 
  "msg": {
    "build_id": "32.20201111.20.0", 
    "stream": "testing-devel", 
    "basearch": "x86_64", 
    "build_dir": "https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/32.20201111.20.0/x86_64", 
    "state": "FINISHED", 
    "result": "SUCCESS"
  }
}