Stackable Release 24.3

[!IMPORTANT] Important dates:

Wednesday, 6th March - Release Planning

Monday, 11th March - cutoff date, begin release

Wednesday, 20th March - there is a customer request for images to be available if possible

Friday, 29th March: Public Holiday for many

Monday, 31st March: Public Holiday for many

[!IMPORTANT] Release blockers:

[x] bug: HDFS env overrides: https://github.com/stackabletech/hdfs-operator/issues/497

[x] Feature: Trino 442 (was just released)

[ ] Feature/Security issue: Zookeeper 3.9.2. Revert the revert. Remove 3.9.1 from the conf.py

Known issues

[x] bug: Trino config overrides don't work as expected. Not part of this release.
[x] regression: https://github.com/stackabletech/issues/issues/547. There is a workaround. We decided not to address it at all before the release.

Release checklists

### General pre-requisites (before feature freeze)
- [ ] https://github.com/stackabletech/issues/issues/537
- [ ] https://github.com/stackabletech/issues/issues/538
- [ ] https://github.com/stackabletech/issues/issues/539
- [ ] https://github.com/stackabletech/issues/issues/542
- [ ] https://github.com/stackabletech/issues/issues/543
- [x] Test SDP release upgrade against several demos (i.e. install release, run demo, bump release, check what needs to be patched) @NickLarsenNZ
- [ ] https://github.com/stackabletech/issues/issues/544
- [ ] https://github.com/stackabletech/issues/issues/502

### Other release-specific pre-requisites
- [ ] https://github.com/stackabletech/issues/issues/521
- [ ] https://github.com/stackabletech/kafka-operator/issues/667
- [ ] https://github.com/stackabletech/documentation/issues/549

Feature freeze

This will not be so crucial with release branches, but is nonetheless sensible as it will make it easier to cherry-pick any release-related bugfixes from main into the release branch.

End of the release cycle (Release day)

#### Technical tasks
- [x] Create release branches for docker-images (see stackable-utils for script to create branches)
- [x] Create release tag(s) for docker-images (see stackable-utils for scripts to create tags)
- [x] Create release branches for operators (see stackable-utils for script to create branches)
- [x] Create release tag(s) for operators (see stackable-utils for scripts to create tags)
- [x] Update changelogs in main branches (see stackable-utils for script to do this)
- [x] Create release tag for stackable-cockpit (optional, highly experimental, requires manual tag creation)
- [ ] https://github.com/stackabletech/issues/issues/551
- [x] Check getting started scripts (use a table in Nuclino)  https://app.nuclino.com/Stackable/Engineering/Release-243-d27e6f92-b437-4d86-9aa5-453a4b4a1148
- [x] Check (new) stackablectl Stacks and Demos (use a table in Nuclino)  https://app.nuclino.com/Stackable/Engineering/Release-243-d27e6f92-b437-4d86-9aa5-453a4b4a1148
- [x] Test with locally updated (to new release number) `releases.yaml`
- [x] Update `release.yaml` : https://github.com/stackabletech/release/pull/27
- [x] Check that an upgrade can be performed on an existing cluster without data loss.
- [x] ~~Run all the test suites~~

#### Documentation tasks
- [ ] https://github.com/stackabletech/issues/issues/549
- [x] Upgrade guide: Document how to use stackablectl to uninstall all and install new release
- [x] Upgrade guide: Document how to use helm to uninstall all and install new release
- [x] Upgrade guide: Every breaking change of all our operators
- [x] Upgrade guide: List dropped supported product versions (if there are some)
- [x] Upgrade guide: List dropped supported operators (if there are some)
- [x] Upgrade guide: List supported k8s versions
- [x] Update version of main documentation repo (@fhennig can assist)
- [x] Set the release to "Released" in the Feature Tracker and create a new release
- [x] Update the getting-started page in the main docs and check it works with this release: https://github.com/stackabletech/documentation/blob/main/modules/ROOT/pages/getting_started.adoc
- [x] Add a note about the upcoming (but unknown) TLS CRD changes
- [ ] https://github.com/stackabletech/demos/pull/24

Marketing tasks can now reference published documentation.

#### Marketing tasks
- [x] Write marketing / customer oriented release summary to be published in the marketing channels
- [x] Update the homepage banner (as long as we have it) to point to the new release
- [x] Write a blogpost / news article announcing the new release (optional)
- [ ] ~~Write a description of new demos for homepage/demos section~~ No new demos in this release.
- [x] Announce Release on linkedIn
- [x] Announce Release in Newsletter (optional)
- [x] Post an announcement in the GitHub [Discussions Announcement forum](https://github.com/stackabletech/community/discussions/categories/announcements) and make it a pinned discussion while at the same time removing the old pinned thread @adwk67: draft is here: https://stackable-workspace.slack.com/archives/C06N4P142UA/p1711116694997849
- [x] Post an announcement in Discord
- [x] Post an announcement on DOK Community in the #be-shameless Channel (Ping Lars or Jim)
- [x] Post an announcement via OSBA (Ping Lars)
- [x] Send announcement to Kubernetes Podcast (Ping Lars)
- [x] Send announcement to Heise
- [x] Ping the stackable-ionos-tech channel or anyone responsible once all tags are created

### Post-release tasks
- [x] Bump Rust version. This can be done [in this file](https://github.com/stackabletech/operator-templating/blob/87a6d03e65bf137053a3fea1a051d5951e6f6853/config/rust.yaml#L5) by changing `rust_version` and also for the ubi base image [here](https://github.com/stackabletech/docker-images/blob/main/ubi8-rust-builder/Dockerfile#L25). Please be aware that this action will change it for all repositories at the same time (when merging the templating PRs).
- [x] Doesn't work. Renovate is not automatable ~~Run renovate manually by starting this action: https://github.com/stackabletech/ci/blob/main/.github/workflows/renovate.yml~~
- [x] Check/bump versions of kube-rs and k8s-openapi (also checking the version of k8s we build against)
- [x] Check/bump ubi8 base image
- [x] preflight now checks automatically it's own version and only runs if latest ~~Check/bump preflight~~
- [ ] https://github.com/stackabletech/issues/issues/556
- [x] ~~Define product versions to include in the next release~~ We have a meeting planned.
- [ ] https://github.com/stackabletech/issues/issues/570

Test SDP release upgrade against several demos (i.e. install release, run demo, bump release, check what needs to be patched)

I tested the following operator upgrades:

[x] airflow-scheduled-job
[x] trino-taxi-data
[x] hbase-hdfs-load-cycling-data (breaking changes documented in https://github.com/stackabletech/documentation/pull/565)

detailed steps

```sh # Fesh cluster (kind delete cluster 2>/dev/null && kind create cluster 2>/dev/null) # Install a release stackablectl release in 23.11 # Check that we are definitely running the correct images kubectl get pods -n stackable-operators -o yaml | yq -y '.items[] | .status.containerStatuses[] | .image' # Install a demo stackablectl demo in airflow-scheduled-job # and trino-taxi-data, hbase-hdfs-load-cycling-data # Double check image versions kubectl get pods -n default -o yaml | yq -y '.items[] | .status.containerStatuses[] | .image' # Uninstall the old release stackablectl release uninstall 23.11 # Uninstall the CRDS kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/main/deploy/helm/airflow-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/main/deploy/helm/commons-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/main/deploy/helm/druid-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/main/deploy/helm/hbase-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/main/deploy/helm/hdfs-operator/crds/crds.yaml # kubectl replace -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/main/deploy/helm/hello-world-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/main/deploy/helm/hive-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/main/deploy/helm/kafka-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/main/deploy/helm/listener-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/main/deploy/helm/nifi-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/main/deploy/helm/opa-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/main/deploy/helm/secret-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/main/deploy/helm/spark-k8s-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/main/deploy/helm/superset-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/main/deploy/helm/trino-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/main/deploy/helm/zookeeper-operator/crds/crds.yaml # Install the dev release stackablectl release install dev # Check that we are definitely running the correct images kubectl get pods -n stackable-operators -o yaml | yq -y '.items[] | .status.containerStatuses[] | .image' # Double check image versions, filter for running images as old completed Spark jobs will show the old image kubectl get pods -n default -o yaml | yq -y '.items[] | .status.containerStatuses[] | select(.started == true) | .image' # Check operator logs for errors kubectl logs -n stackable-operators -l stackable.tech/vendor=Stackable --follow --max-log-requests=16 ```

stackabletech / issues

Release 24.3 #488

Stackable Release 24.3

Known issues

Release checklists

Feature freeze

End of the release cycle (Release day)