Test all demos on ARM

[fhennig]

We already did some work on https://github.com/stackabletech/demos/tree/spike/demos-on-arm which leaves just the demos open not touched right now.

Shouldn't be to much. The test would be to have a ARM64 cluster and see if the demos work as expected (Be aware that it will only work for 23.11 and stackable-experimental contains ARM64 images and not multi arch for it)

[adwk67]

General comments

• add something to the docs about using mixed-arch clusters (e.g. set nodeSelector for daemonsets, pods, jobs etc.)
• stacklet endpoints do not work ~for some of the demos~

Demos

• :heavy_check_mark: airflow-scheduled-job: corrected the images: https://github.com/stackabletech/demos/commit/b13ae6355dab787452a4fea0f0f83a679e3c4624, https://github.com/stackabletech/demos/commit/5f11fbab08de8363643c0183f7294d50cba7dc83
• :heavy_check_mark: hbase-hdfs-load-cycling-data
• :heavy_check_mark: nifi-kafka-druid-earthquake-data: https://github.com/stackabletech/demos/commit/5f11fbab08de8363643c0183f7294d50cba7dc83
• :heavy_check_mark: nifi-kafka-druid-water-level-data: https://github.com/stackabletech/demos/commit/5f11fbab08de8363643c0183f7294d50cba7dc83
• :x: spark-k8s-anomaly-detection-taxi-data: possibly fixable, requires docker.stackable.tech/demos/pyspark-k8s-with-kafka-and-iceberg:3.3.0-stackable0.2.0 as arm64
• :heavy_check_mark: trino-iceberg: some trino queries work, but connectivity is unstable (maybe cluster-related) e.g.
  • Query 20240308_091808_00028_q7hxj failed: Failed to query OPA backend
  • Query 20240308_090407_00024_q7hxj failed: Failed connecting to Hive metastore
• :heavy_check_mark: trino-taxi-data: https://github.com/stackabletech/demos/commit/6a6d7834bf5f09253681a778c07f8e00e79e2cf0
• :x: jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data: possibly fixable, requires docker.stackable.tech/demos/pyspark-k8s-with-scikit-learn:3.3.0-stackable23.4 as arm64
• :heavy_check_mark: logging
• :x: signal-processing: timescale chart does not seem to be multi-arch and is no longer maintained (may be fixable using a postgres operator instead)
• :x: question: data-lakehouse-iceberg-trino-spark: possibly fixable, requires docker.stackable.tech/demos/pyspark-k8s-with-kafka-and-iceberg:3.3.0-stackable0.2.0 as arm

[sbernauer]

@adwk67 would you be willing to switch docker.stackable.tech/demos/pyspark-k8s-with-kafka-and-iceberg:3.3.0-stackable0.2.0 to pull the deps using spec.deps.packages similar to the following? This way it should (tm) work (and a thing we want anyway I guess :) )

Using Spark 3.5 something like:

  deps:
    packages:
      - org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.4.3
      - org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.0

[adwk67]

As discussed today, closing this with fix issues carried over into https://github.com/stackabletech/demos/issues/25 and https://github.com/stackabletech/demos/issues/26.