Closed maltesander closed 1 month ago
:green_circle: airflow-scheduled-jobs
Upgrade 23.4 -> 23.7 worked.
Had to uninstall postgres and redis (new versions) manually
# Clone demos repository and checkout release-24.3 branch (/tmp)
git clone git@github.com:stackabletech/demos.git
git checkout release-24.3
# Clone release and use 24.3
git clone git@github.com:stackabletech/release.git
# point local stacks file to local folder
stackablectl demo in airflow-scheduled-job --demo-file /tmp/demos/demos/demos-v2.yaml --stack-file /tmp/demos/stacks/stacks-v2.yaml --release-file /tmp/release/releases.yaml
# uninstall release
stackablectl release uninstall 24.3
# replace crds
kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/release-24.7/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/release-24.7/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/release-24.7/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/release-24.7/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/release-24.7/deploy/helm/spark-k8s-operator/crds/crds.yaml
# postgres gets a new version: 6: release postgresql-airflow (13.2.18) already installed, skipping requested version 15.5.16
helm uninstall postgresql-airflow
# redis gets a new version:
helm uninstall redis-airflow: 6: release redis-airflow (18.1.6) already installed, skipping requested version 19.6.1
# will use 24.7 release and demo / stack files
stackablectl demo in airflow-scheduled-job
:green_circle: hbase-hdfs-load-cycling-data
# Clone demos repository and checkout release-24.3 branch (/tmp)
git clone git@github.com:stackabletech/demos.git
git checkout release-24.3
# Clone release and use 24.3
git clone git@github.com:stackabletech/release.git
# point local stacks file to local folder (and adapt files to be releative)
stackablectl demo in hbase-hdfs-load-cycling-data --demo-file /tmp/demos/demos/demos-v2.yaml --stack-file /tmp/demos/stacks/stacks-v2.yaml --release-file /tmp/release/releases.yaml
stackablectl release uninstall 24.3
# update crds
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/release-24.7/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/release-24.7/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/release-24.7/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/release-24.7/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/release-24.7/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/release-24.7/deploy/helm/zookeeper-operator/crds/crds.yaml
# copy the secret `secret-provisioner-tls-ca` to the operator's namespace
kubectl get secrets secret-provisioner-tls-ca --output=yaml | \
sed 's/namespace: .*/namespace: stackable-operators/' | \
kubectl create --filename=-
# install 24.7
stackablectl demo in hbase-hdfs-load-cycling-data
Results in errors in the operator, but works after deleting the statefulsets:
2024-07-25T09:46:58.862709Z ERROR hdfs_controller: stackable_operator::logging::controller: Failed to reconcile object controller.name="hdfsclusters.hdfs.stackable.tech" error=reconciler for object HdfsCluster.v1alpha1.hdfs.stackable.tech/hdfs.default fa
iled error.sources=[cannot create role group stateful set "hdfs-namenode-default", failed to apply patch, unable to patch resource "hdfs-namenode-default", ApiError: StatefulSet.apps "hdfs-namenode-default" is invalid: spec: Forbidden: updates to statefu
lset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden: Invalid (ErrorResponse { status: "Failure", message: "StatefulSet.apps \"hdfs-namenode-defau
lt\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden", reason: "Invalid", code: 422 }), Statef
ulSet.apps "hdfs-namenode-default" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden: Invalid]
3.3.6 -> 3.4.0 (not supported at the moment, so expected)
namenode File system image contains an old layout version -66.
namenode An upgrade to version -67 is required.
namenode Please restart NameNode with the "-rollingUpgrade started" option if a rolling upgrade is already started; or restart NameNode with the "-upgrade" option to start a new upgrade.
# Reapplied jobs fail (folders existing etc.)
EDIT: Installing from main / 24.7 leads to failing distcp job (due to removing map reduce from docker image)
Error: Unable to initialize main class org.apache.hadoop.tools.DistCp
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/Job
:green_circle: end-to-end-security
No upgrade required. But requires https://github.com/stackabletech/demos/pull/67 for krb5 version (1.21.1) fix.
Works with that fix.
:green_circle: nifi-kafka-druid-earthquake-data :green_circle: nifi-kafka-druid-water-level-data
# Clone demos repository and checkout release-24.3 branch (/tmp)
git clone git@github.com:stackabletech/demos.git
# switch branch to 24.3
git checkout release-24.3
# Clone release and use 24.3
git clone git@github.com:stackabletech/release.git
# point local stacks file to local folder (and adapt files to be releative instead then github raw)
stackablectl demo in nifi-kafka-druid-earthquake-data --demo-file /tmp/demos/demos/demos-v2.yaml --stack-file /tmp/demos/stacks/stacks-v2.yaml --release-file /tmp/release/releases.yaml
# uninstall operators
stackablectl release uninstall 24.3
# update crds
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/release-24.7/deploy/helm/commons-operator/crds/crds.yaml
kubectl create -f https://raw.githubusercontent.com/stackabletech/listener-operator/release-24.7/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/release-24.7/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/release-24.7/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/release-24.7/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/release-24.7/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/release-24.7/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/release-24.7/deploy/helm/zookeeper-operator/crds/crds.yaml
# 6: release postgresql-superset (13.2.18) already installed, skipping requested version 15.5.16
# 6: release postgresql-druid (13.2.18) already installed, skipping requested version 15.5.16
# 6: release postgresql-superset (13.2.18) already installed, skipping requested version 15.5.16
helm upgrade minio minio/minio --version 5.2.0
helm upgrade postgresql-druid bitnami/postgresql --version 15.5.16
helm upgrade postgresql-superset bitnami/postgresql --version 15.5.16
# replace druid.yaml in stack for fix from PR
stackablectl demo in nifi-kafka-druid-earthquake-data --stack-file stacks/stacks-v2.yaml
Requires https://github.com/stackabletech/demos/pull/67 for druid db credentials fix.
🟢 signal-processing
# Clone demos repository and checkout release-24.3 branch (/tmp)
git clone git@github.com:stackabletech/demos.git
git checkout release-24.3
# Clone release and use 24.3
git clone git@github.com:stackabletech/release.git
# point local stacks file to local folder
stackablectl demo in signal-processing --demo-file /tmp/demos/demos/demos-v2.yaml --stack-file /tmp/demos/stacks/stacks-v2.yaml --release-file /tmp/release/releases.yaml
# uninstall release
stackablectl release uninstall 24.3
# replace crds
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/release-24.7/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/release-24.7/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/release-24.7/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/release-24.7/deploy/helm/zookeeeper-operator/crds/crds.yaml
# install latest
stackablectl demo in signal-processing
🟢 logging
# Clone demos repository and checkout release-24.3 branch (/tmp)
git clone git@github.com:stackabletech/demos.git
git checkout release-24.3
# Clone release and use 24.3
git clone git@github.com:stackabletech/release.git
# point local stacks file to local folder
stackablectl demo in logging --demo-file /tmp/demos/demos/demos-v2.yaml --stack-file /tmp/demos/stacks/stacks-v2.yaml --release-file /tmp/release/releases.yaml
# uninstall release
stackablectl release uninstall 24.3
# replace crds
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/release-24.7/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/release-24.7/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/release-24.7/deploy/helm/zookeeper-operator/crds/crds.yaml
# delete setup-opensearch-dashboards job
# uninstall helm releases
helm uninstall vector-aggregator
# install latest
stackablectl demo in signal-processing
🟢 trino-iceberg
release-24.3
)Loosely following: https://github.com/stackabletech/demos/issues/59#issuecomment-2236602426
# Clone demos repository and checkout release-24.3 branch (/tmp)
stackablectl demo install trino-iceberg --demo-file demos/demos-v2.yaml --stack-file stacks/stacks-v2.yaml
# FAILED: docker.stackable.tech/stackable/opa:0.66.0-stackable24.3.0 fails to pull
# Have to edit the stack/demo refs to not use main.
# upgrade postgres chart
helm upgrade postgresql-hive-iceberg bitnami/postgresql --version 15.5.16
helm upgrade minio minio/minio --version 5.2.0
# uninstall operators
stackablectl release uninstall 24.3
# update crds
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/release-24.7/deploy/helm/commons-operator/crds/crds.yaml
kubectl create -f https://raw.githubusercontent.com/stackabletech/listener-operator/release-24.7/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/release-24.7/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/release-24.7/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/release-24.7/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/release-24.7/deploy/helm/opa-operator/crds/crds.yaml
# install 24.7 version of operators
stackablectl demo install trino-iceberg # if changes are needed: --stack-file stacks/stacks-v2.yaml
# If changes are needed, checkout a branch for fixes
git checkout main
git pull
git checkout -b fix/release-24.7-trino-iceberg
stackablectl demo install trino-iceberg --demo-file=demos/demos-v2.yaml --stack-file stacks/stacks-v2.yaml
# If the fixes are needed in main too, cherry pick the relevant ones
...
⚠ tpch.sf5
doesn't appear in DBeaver. I have checked the OPA policies and see nothing that should stop that. I don't think it is worthy of halting the release though.
:green_circle: spark-k8s-anomaly-detection-taxi-data
works with https://github.com/stackabletech/demos/pull/67
# point local stacks file to local folder (and adapt files to be releative instead then github raw)
stackablectl demo in spark-k8s-anomaly-detection-taxi-data --demo-file /tmp/demos/demos/demos-v2.yaml --stack-file /tmp/demos/stacks/stacks-v2.yaml --release-file /tmp/release/releases.yaml
# uninstall operators
stackablectl release uninstall 24.3
# update crds
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/release-24.7/deploy/helm/commons-operator/crds/crds.yaml
kubectl create -f https://raw.githubusercontent.com/stackabletech/listener-operator/release-24.7/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/release-24.7/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/release-24.7/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/release-24.7/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/release-24.7/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/release-24.7/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/release-24.7/deploy/helm/trino-operator/crds/crds.yaml
# to make it work for reinstall
# helm upgrade minio minio/minio --version 5.2.0
# helm upgrade postgresql-hive bitnami/postgresql --version 15.5.16
# helm upgrade postgresql-hive-iceberg bitnami/postgresql --version 15.5.16
# helm upgrade postgresql-superset bitnami/postgresql --version 15.5.16
kubectl delete jobs create-spark-anomaly-detection-job
kubectl delete jobs setup-superset
kubectl delete sparkapplications spark-ad
# use local files in stacks for fixes
stackablectl demo in spark-k8s-anomaly-detection-taxi-data --stack-file stacks/stacks-v2.yaml
``
:green_circle: jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data
Gets stuck on starting the hdfs-datanode-default-0
and hdfs-namenode-default-0
pods. This seems to be a local issue.
Edit Malte: Could not reproduce, works as expected (there was a change from hadoop 3.3.6 to 3.3.4, which maybe made the difference?)
🟢 trino-taxi-data
works with https://github.com/stackabletech/demos/pull/67
# point local stacks file to local folder (and adapt files to be releative instead then github raw)
stackablectl demo in trino-taxi-data --demo-file /tmp/demos/demos/demos-v2.yaml --stack-file /tmp/demos/stacks/stacks-v2.yaml --release-file /tmp/release/releases.yaml
# uninstall operators
stackablectl release uninstall 24.3
# update crds
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/main/deploy/helm/commons-operator/crds/crds.yaml
kubectl create -f https://raw.githubusercontent.com/stackabletech/listener-operator/main/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/main/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/main/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/main/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/main/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/main/deploy/helm/opa-operator/crds/crds.yaml
# to make it work for reinstall (or patch product versions)
# helm upgrade minio minio/minio --version 5.2.0
# helm upgrade postgresql-hive bitnami/postgresql --version 15.5.16
# helm upgrade postgresql-superset bitnami/postgresql --version 15.5.16
kubectl delete jobs create-ny-taxi-data-table-in-trino
kubectl delete jobs setup-superset
kubectl delete jobs load-ny-taxi-data
# use local files in stacks for fixes
stackablectl demo in trino-taxi-data --stack-file stacks/stacks-v2.yaml
🟢 data-lakehouse-iceberg-trino-spark
During install:
Error from server (BadRequest): error when creating "/tmp/manifest/spark-ingestion-job.yaml": SparkApplication in version "v1alpha1" cannot be handled as a SparkApplication : strict decoding error: unknown field "spec.version"
in create-spark-ingestion-job
container (24.3 release branch of demos) -> remove spec.version
from SparkApplication definition in create-spark-ingestion-job-manifest
configmapDuring upgrade:
DEBUG Unsuccessful: ErrorResponse { status: "Failure", message: "Apply failed with 1 conflict: conflict with \"kubectl-edit\" using trino.stackable.tech/v1alpha1: .spec.image.productVersion", reason: "Conflict", code: 409 }
-> manually editing the trino version worked fineDEBUG Unsuccessful: ErrorResponse { status: "Failure", message: "Apply failed with 1 conflict: conflict with \"kubectl-edit\" using v1: .data.spark-ingestion-job.yaml", reason: "Conflict", code: 409 }
-> the first change (spec.version) was correctly applied, only at the second one (spec.sparkImage.productVersion) it failed (similar to the trino one above) -> manual edit to fixAfter upgrade:
ValueError: A template named LakehouseKafkaIngest already exists.
-> not an actual release relevant errorfor: "/tmp/manifest/spark-ingestion-job.yaml": error when patching "/tmp/manifest/spark-ingestion-job.yaml": sparkapplications.spark.stackable.tech "spark-ingest-into-lakehouse" is forbidde n: User "system:serviceaccount:default:demo-serviceaccount" cannot patch resource "sparkapplications" in API group "spark.stackable.tech" in the namespace "default"
-> even though the changes above to spark-ingestion-job.yaml
were applied, the create-spark-ingestion-job
couldn't edit the SparkApplication -> manual edit there as well to fix# checkout repos temporarily in tmp folder
git clone git@github.com:stackabletech/demos.git
git checkout release-24.3
git clone git@github.com:stackabletech/release.git
# adapt files to be relative and point to local folders instead of using github raw
# point to those local files during install of demo
# add listener-operator to data-lakehouse stack
stackablectl demo in data-lakehouse-iceberg-trino-spark --demo-file demos/demos/demos-v2.yaml --stack-file demos/stacks/stacks-v2.yaml --release-file release/releases.yaml
# list of errors while installing this see above
# uninstall operators
stackablectl release uninstall 24.3
# update crds
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/main/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/main/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/main/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/main/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/main/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/main/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/main/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/main/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/main/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/main/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/main/deploy/helm/zookeeper-operator/crds/crds.yaml
# to make it work for reinstall (or patch product versions)
helm upgrade minio minio/minio --version 5.2.0
helm upgrade postgresql-hive bitnami/postgresql --version 15.5.16
helm upgrade postgresql-superset bitnami/postgresql --version 15.5.16
helm upgrade postgresql-hive-iceberg bitnami/postgresql --version 15.5.16
# install latest
stackablectl demo in data-lakehouse-iceberg-trino-spark
# list of errors while upgrading see above
Description
For each demo: test the upgrading process from 24.3 to 24.7 release and document necessary changes for upgrading if necessary. Also check the steps in the demo documentation to ensure the proper functionality of the demo with the 24.7 release.
PR for changes