next: new release on 2021-09-28 (35.20210924.1.0)

[dustymabe]

First, verify that you meet all the prerequisites

Name this issue next: new release on YYYY-MM-DD with today's date. Once the pipeline spits out the new version ID, you can append it to the title e.g. (31.20191117.2.0).

Pre-release

Promote next-devel changes to next

• [x] Merge https://github.com/coreos/fedora-coreos-streams/pull/371
• [x] Add the ok-to-promote label to the issue
• [x] Review the promotion PR opened by the bot against the next branch on https://github.com/coreos/fedora-coreos-config
• [x] Once CI has passed, merge it

Build

• [x] Start a pipeline build (select next, leave all other defaults)
• [x] Post a link to the jobs as a comment to this issue
  • [x] x86_64
  • [x] aarch64 (multi-arch pipeline)
• [x] Wait for the job to finish and succeed
  • [x] x86_64
  • [x] aarch64

Sanity-check the build

Using the the build browser for the next stream:

• [x] Verify that the parent commit and version match the previous next release (in the future, we'll want to integrate this check in the release job)
  • [x] x86_64
  • [x] aarch64
• [x] Check kola AWS runs to make sure they didn't fail
  • [x] x86_64
  • [x] aarch64
• [x] Check kola GCP run to make sure it didn't fail
• [x] Check kola OpenStack run to make sure it didn't fail

⚠️ Release ⚠️

IMPORTANT: this is the point of no return here. Once the OSTree commit is imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Run the release job

• [x] Run the release job, filling in for parameters next and the new version ID
• [x] Post a link to the job as a comment to this issue
• [x] Wait for job to finish

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh metadata (stream and updates)

From a checkout of this repo:

• [x] Update stream metadata, by running:

fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/next/releases.json  -output-file=streams/next.json -pretty-print

• [x] Add a rollout. For a 48-hour rollout starting at 10 AM ET, run:

./rollout.py add next <version> "10 am ET" 48

• [x] Commit the changes and open a PR against the repo. Paste the output of make print-rollouts into the PR description.
• [x] Post a link to the PR as a comment to this issue
• [x] Wait for the PR to be approved.
• [x] Once approved, merge it and verify that the sync-stream-metadata job syncs the contents to S3
• [x] Verify the new version shows up on the download page
• [x] Verify the incoming edges are showing up in the update graph.
  • [x] x86_64
  • [x] aarch64

Update graph manual check

``` curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=next&rollout_wariness=0' curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=aarch64&stream=next&rollout_wariness=0' ```

NOTE: In the future, most of these steps will be automated.

Housekeeping

• [ ] If one doesn't already exist, open an issue in this repo for the next release in this stream. Use the approximate date of the release in the title.
• [ ] Issues opened via the previous link will automatically create a linked Jira card. Assign the GitHub issue and Jira card to the next person in the rotation.

[dustymabe]

This will be the first next release based on F35 content (now that F35 beta is GO to be shipped next week).

[dustymabe]

just did a rebase from testing to next-devel and these are the interesting bits..

Downgraded:
  bind-libs 32:9.16.20-3.fc34 -> 32:9.16.20-2.fc35
  bind-license 32:9.16.20-3.fc34 -> 32:9.16.20-2.fc35
  bind-utils 32:9.16.20-3.fc34 -> 32:9.16.20-2.fc35
  bsdtar 3.5.2-2.fc34 -> 3.5.2-1.fc35
  btrfs-progs 5.14-2.fc34 -> 5.13.1-1.fc35
  conmon 2:2.0.29-2.fc34 -> 2:2.0.29-0.2.dev.git2947bb2.fc35
  container-selinux 2:2.167.0-1.fc34 -> 2:2.164.2-1.fc35
  cracklib 2.9.6-27.fc34 -> 2.9.6-26.fc35
  cracklib-dicts 2.9.6-27.fc34 -> 2.9.6-26.fc35
  crun 1.0-1.fc34 -> 0.21-1.fc35
  dnsmasq 2.86-1.fc34 -> 2.85-6.fc35
  efi-filesystem 5-4.fc34 -> 5-3.fc35
  ethtool 2:5.14-1.fc34 -> 2:5.13-2.fc35
  fuse-overlayfs 1.7.1-2.fc34 -> 1.7.1-1.fc35
  libarchive 3.5.2-2.fc34 -> 3.5.2-1.fc35
  libuv 1:1.42.0-2.fc34 -> 1:1.41.1-2.fc35
  openssl 1:1.1.1l-1.fc34 -> 1:1.1.1k-2.fc35
  openssl-libs 1:1.1.1l-1.fc34 -> 1:1.1.1k-2.fc35
  ostree 2021.4-2.fc34 -> 2021.3-2.fc35
  ostree-libs 2021.4-2.fc34 -> 2021.3-2.fc35
  podman 3:3.3.1-1.fc34 -> 3:3.3.0-1.fc35
  podman-plugins 3:3.3.1-1.fc34 -> 3:3.3.0-1.fc35
  skopeo 1:1.4.1-1.fc34 -> 1:1.4.0-2.fc35
  slirp4netns 1.1.12-2.fc34 -> 1.1.12-1.fc35
  vim-minimal 2:8.2.3404-1.fc34 -> 2:8.2.3367-1.fc35
Removed:
  compat-readline5-5.2-39.fc34.x86_64
  iptables-1.8.7-3.fc34.x86_64
  libtextstyle-0.21-4.fc34.x86_64
  libusbx-1.0.24-2.fc34.x86_64
  systemd-rpm-macros-248.7-1.fc34.noarch
Added:
  dracut-squash-055-3.fc35.1.x86_64
  gdbm-libs-1:1.20-2.fc35.x86_64
  iptables-legacy-libs-1.8.7-13.fc35.x86_64
  libusb1-1.0.24-4.fc35.x86_64
  squashfs-tools-4.5-2.fc35.x86_64
  systemd-resolved-249.4-1.fc35.x86_64
  util-linux-core-2.37.2-1.fc35.x86_64

I'm thinking we don't really want any Downgraded packages so we should proably massage our package lists to fix that.

[dustymabe]

• ~x86_64 pipeline build~
• ~aarch64 pipeline build~

[dustymabe]

New promotion PR: https://github.com/coreos/fedora-coreos-config/pull/1240

We wanted the new glib2 (in F35 beta) and also iptables-legacy.

[dustymabe]

• x86_64 pipeline build
• aarch64 pipeline build

[dustymabe]

• kola-aws#612 (x86_64) - ✔️
• kola-openstack#370 (x86_64) - ✔️
• kola-gcp#570 (x86_64) - ⚠️
  • job was aborted but tests passed - ✔️

[dustymabe]

• kola-aws#617 (aarch64) ✔️

[dustymabe]

• Release Job: fedora-coreos-pipeline-release#727

[dustymabe]

• Release Job: fedora-coreos-pipeline-release#727

That one failed because something else was updating the OSTree repo at the same time:

2021-09-28 11:06:18,025 ERROR builtins - COMMAND: ['ostree', '--repo=/mnt/koji/ostree/repo', 'summary', '-u']
--
2021-09-28 11:06:18,025 ERROR builtins -  STDERR: error: Locking repo exclusive failed: Resource temporarily unavailable

New run in: fedora-coreos-pipeline-release#728

[dustymabe]

• https://github.com/coreos/fedora-coreos-streams/pull/378

[dustymabe]

No Housekeeping to do for this release.