ELN and Sidetags - Tracker

[tdawson]

ELN has to deal with several issues dealing with package ordering. In Fedora, the approved way of dealing with complicated package ordering is to use side-tags. We plan on implementing side-tags in ELN to deal with package ordering.

This issue to to track the various issues related to ELN side-tags and package ordering.

[bookwar]

The option to create sidetags for the dedicated base tag is already available

But we need an agreement with Koji admins on the policy

Currently it fails with:

$ fedpkg request-side-tag --base-tag eln
Could not execute request_side_tag: policy violation (sidetag)

[bookwar]

Sidetag permissions requested in https://pagure.io/fedora-infrastructure/issue/9329

[bookwar]

Proposed workflow:

1) trigger on bodhi update being promoted to stable, rather than on Koji tagging event 2) get the full list of packages from the Bodhi update 3) sort them by time of build 4) request eln sidetag 5) walk through the list of packages rebuilding them one by one in the sidetag 6) if one of the builds fails - stop and report the error 7) if all packages are built successfully, merge the sidetag to ELN

The proposed flow won't fully cover the bootstrapping case, but can be extended with it later

[contyk]

A couple of comments:

1. Triggering on bodhi updates sounds good as it might also imply some test coverage; with Rawhide these are generally pretty fast, even though it will add some delay.
2. Ok!
3. You could be smarter here and see what builds finished before other builds started and put those into buckets for parallel build submission. This would generally improve the step 5; by a lot (see below).
4. Ok!
5. Note you need to wait-repo between builds to ensure the previously built package is in your buildroot. Just waiting for the build to finish isn't enough. And kojira runs take a while. Modules struggle with the same thing with their buildorder handling but it cannot be avoided. Some (potentially many of those, think gcc mass rebuilds) only need a handful of packages built in order and the rest (potentially thousands of packages) can be submitted in parallel as the order doesn't matter. If you split this into buckets correctly in step 3, this should be a breeze.
6. Sounds good to me. You should probably also cancel all running builds in your bucket to save resources as you're failing the whole task anyway.
7. Ok!

You could cover bootstrap by changing steps 2 and 3; if you know the side tag used, you can list its tagging history to determine if some packages were built more than once. You can then query those builds to get their SCMURLs and build those, in the same order. Again, inspecting the start/end times to determine what goes into what bucket is crucial.

Also CC @sgallagher; he's also dealth with this already.

[hroncok]

A random idea: A side tag could be constructed for each build. It is possible to query the rawhide / side-tag builds for the buildroot content with koji list-buildroot <buildroot_id> (or an API query equivalent). For each RPM in the buildroot, it is possible to query the Koji build (e.g. with koji rpminfo tk-8.6.10-3.fc32.x86_64) and based on that build's nevr and/or SCM commit hash, appropriate action can be selected (wait for it / tag the rawhide build / tag the eln build). That seems a little complicated and possibly slow, OTOH could prevent all the "this was built in different order than in Fedora" problems (not only when side tags are involved).

[contyk]

It is an idea worth considering but indeed probably quite slow. In extreme cases Rawhide builds could be triggering ELN builds faster than we could process them with all this overhead.

And, of course, it further delays RHEL builds (which are triggered by ELN builds); while I believe the whole cycle is plenty quick now, some still consider it slow.

Anyway, not dismissing, it's an option. It could made some parts of the sync simpler even.

[bookwar]

Saving it from local conversation:

The current workaround how to land build group in Rawhide/ELN. Needs to be implemented manually by the Fedora maintainer.

1. Request dynamic sidetag for rawhide and build all required packages into it with fedpkg request-side-tag
2. if all rawhide packages were built successfully, request dynamic sidetag for eln with fedpkg request-side-tag --base-tag eln and build the same packages in the same order into it.
3. If eln builds also succeed, create bodhi update for rawhide with bodhi updates new --from-tag <fedora-sidetag>
4. Wait until it passes CI and Gating and builds land into rawhide tag
5. Create bodhi update for eln builds with bodhi updates new --from-tag <eln-sidetag>

[hroncok]

Just curious. Why not eln bodhi update in the last step?

[bookwar]

I was thinking by some reason that we bypass bodhi in ELN. Which is not true https://bodhi.fedoraproject.org/releases/ELN

You are right. bodhi update should be good for ELN too. I'll update it

[bookwar]

Examples of a message Bodhi sends when bodhi update is ready to test

https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete&rows_per_page=1&delta=127800

{
  "username": "amqp-bridge", 
  "source_name": "datanommer", 
  "certificate": "...", 
  "i": 1242191, 
  "timestamp": 1613771559.0, 
  "msg_id": "2021-d9b2e53a-bc63-4539-886e-c051469199f3", 
  "crypto": "x509", 
  "topic": "org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete", 
  "headers": {}, 
  "signature": "x3PcqWfgMD+CPzV8vQVjPwJGpr6LoQeNSrgIn05bfuC+thrqpAM98vSMy0CAfOD7tcuSZf0rDLs5\nXJgQJPs2SmXavX6X2ciOik4vUQMjJI94KT08MRJI3wOtfy4PiMLX4JOBWPjNzxWa/aoAvwB+jitL\ns1I9qhxepIrntbKfVhIMVCKDmwJDhLEXPhygszmAtSG6S+flsaRmldtpOcynvrIXbplKF0F1bDm5\nbkFcyaIgaBxOFTzcYGeSPxdOB6JHc3A2nmeJiW4IPEn/Nipc1Wv9D11TxBLv/c4qLSuQ/59EA0rM\n7ZHIkycrjpHQiecdlfuSu4bw+raKuSt/cyZ+gQ==\n", 
  "source_version": "0.9.0", 
  "msg": {
    "re-trigger": false, 
    "artifact": {
      "release": "f35", 
      "type": "koji-build-group", 
      "builds": [
        {
          "nvr": "gdk-pixbuf2-2.42.2-1.fc35", 
          "task_id": 62316027, 
          "scratch": false, 
          "component": "gdk-pixbuf2", 
          "type": "koji-build", 
          "id": 1712686, 
          "issuer": "kalev"
        }, 
        {
          "nvr": "gdk-pixbuf2-xlib-2.40.2-2.fc35", 
          "task_id": 62312435, 
          "scratch": false, 
          "component": "gdk-pixbuf2-xlib", 
          "type": "koji-build", 
          "id": 1712663, 
          "issuer": "kalev"
        }
      ], 
      "repository": "https://bodhi.fedoraproject.org/updates/FEDORA-2021-36fb4782c9", 
      "id": "FEDORA-2021-36fb4782c9-54dbcfdbbd6f26679e090cc57a987dc20041befd"
    }, 
    "contact": {
      "docs": "https://docs.fedoraproject.org/en-US/ci/", 
      "team": "Fedora CI", 
      "email": "admin@fp.o", 
      "name": "Bodhi"
    }, 
    "version": "0.2.2", 
    "agent": "bodhi", 
    "generated_at": "2021-02-19T21:52:39.450389Z"
  }
}

The order of builds can be taken from task ids

This message is sent before the builds pass rawhide gating but after all builds in the sidetag are built successfully. We can trigger the pipeline which will create ELN sidetag and build the same packages in the same order into it. But we shouldn't yet merge this sidetag into ELN.

There should be additional step which is triggered only after bodhi update is pushed to rawhide.

[hroncok]

Alternative solution for site tags:

1. A side tag is merged to Rawhide, it contains N builds from N packages, N >= 1.
2. An ELN side tag is automatically created (if (1) contains at lest one ELN package).
3. Each build from (1) gets tagged directly to the ELN side tag from (2) (yes, the Fedora builds) except packages that are not part of ELN.
4. All builds are koji wait-repoed (including the non-ELN ones that should propagate from rawhide).
5. All ELN packages from (1) are built in the side tag from (2) (any order, parallel, possibly with retries, but the same commits are used, not rawhide HEAD)
6. If the success rate is bigger than a defined constant, all remaining Fedora builds tagged into it (if any) are untagged and the side tag (2) is merged to ELN (e.g. trough bodhi), otherwise human investigates.

[sgallagher]

5. All ELN packages from (1) are built in the side tag from (2) (any order, parallel, but the same commits are used, not rawhide HEAD)
6. If the success rate is bigger than a defined constant, the side tag (2) is merged to ELN, otherwise human investigates

Assuming step 6 includes ensuring we don't tag any of the Fedora builds in as part of merging the side-tag, that is a good idea.

[hroncok]

Right. Edited.

[hroncok]

Another note: If another rawhide side tag happens and the set of the packages is overlapping, I would not recommend processing the second side tag before the first one is merged/deleted.

[sgallagher]

I was assuming we'd be triggering this on the rawhide side-tag getting merged. So unless they merge them together...

[hroncok]

Not together.

1. Rawhide side tag with packages A B C D is merged.
2. ELN creates side tags, starts builds, package D builds for very long time...
3. Meanwhile, a new rawhide side tag with packages B C E F is merged.

[sgallagher]

Ohhh, I misunderstood. Thanks.

[voxik]

As a stopgap solution, can we stop importing builds where there is the slightest chance that their rebuild will be not the right one, let these situation to be handled manually?

[msrb]

I know jkaluza is working on proper side-tag rebuilds for c9s, so maybe the same solution could be applied here.

[tdawson]

I hope so, or at least some variation of it. The problem with the c9s side-tags is that they are manual, and the ELN one's have to be automatic. But, the c9s side-tags have to automatically bring those side-tags into the RHEL build infrastructure. So I think whatever automation they do there, should (hopefully) directly apply here.

[sgallagher]

This should be resolved now that we have changed over to the DistroBuildSync-based rebuilds.