Run/check demos with dev release and main branch and create draft PR for release-related changes

NickLarsenNZ commented 8 months ago

Test the demos against nightly images by using the dev release in stackablectl.

Edit stacks/stacks-v2.yaml and set stackableRelease: dev for each stack.
Deploy the demo, ensure there are no issues (fix anything like CRD changes)
Run through the demo text and make sure it still works/makes sense.

Refactors captured in https://github.com/stackabletech/demos/pull/24

Demos

[!IMPORTANT] To aid in distributing the workload, please put your github handle against the one you will run. For any fixes, drop a link to the PR next to the item.

[x] airflow-scheduled-job (@NickLarsenNZ) - errors getting DAG run logs :disappointed: new UI screenshots needed
[x] data-lakehouse-iceberg-trino-spark (@NickLarsenNZ) - can't get it running :grimacing: @razvan the real time data sources have changed and the Nifi workflow needs to be updated. This affects all versions.
[x] hbase-hdfs-load-cycling-data (@NickLarsenNZ)
[x] jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data (@NickLarsenNZ) - Todo: Spark 3.5.1 image uses Scala 2.13, jupyter/pyspark-notebook:python-3.9 / 3.11 both use Scala 2.12
[x] logging (@NickLarsenNZ)
[x] nifi-kafka-druid-earthquake-data (@adwk67) - also tested with zookeeper 3.9.2
[x] nifi-kafka-druid-water-level-data (@adwk67) - everything works, checked superset/nifi/druid. Minio shows no buckets (data not flushed)
[x] signal-processing (@NickLarsenNZ) - Nifi 1.25, updated Grafana and Jupyterlab
[x] spark-k8s-anomaly-detection-taxi-data (@adwk67). Incompatible scikit-learn version: updated to 1.4.0.
[x] trino-taxi-data (@NickLarsenNZ) - Trino 442, updated rego rules and screenshots.

[!NOTE] Use the local demo and stack files in case you need to modify them in any way:
stackablectl -s stacks/stacks-v2.yaml -d demos/demos-v2.yaml demo install ...

NickLarsenNZ commented 8 months ago

Notes for future me:

# go to the demos repo, generate the list of demos as checkbox items to paste into the issue
find demos -mindepth 1 -maxdepth 1 -type d -printf '%P\n' | sed -e 's/^/- [ ] /' | xsel -b

NickLarsenNZ commented 8 months ago

data-lakehouse-iceberg-trino-spark

I had to create a kind cluster with more nodes:

# three node (two workers) cluster config
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
- role: worker

Then

kind create cluster --name k8s-playground --config kind-config.yaml

I then made local changes to product and image versions, and set the stacks/demos to use local file paths...

The demo install hangs after the minio installation:

❯ stackablectl -s stacks/stacks-v2.yaml -d demos/demos-v2.yaml demo install data-lakehouse-iceberg-trino-spark
  ERROR  Go wrapper function go_install_helm_release encountered an error: failed post-install: 1 error occurred:
        * timed out waiting for the condition

    at src/helm.rs:285

An unrecoverable error occured: demo command error

Caused by these errors (recent errors listed first):
 1: failed to install demo
 2: failed to install stack manifests
 3: failed to install Helm release minio
 4: failed to install Helm release
 5: helm error

It seems to continue after running again, so maybe it's ok?

NickLarsenNZ commented 8 months ago

hbase-hdfs-load-cycling-data

[!NOTE] Original changes were in this PR: https://github.com/stackabletech/demos/pull/23. They will me moved to a single PR for all demo changes.

The demo seems to work, just needed to fix the listenerClass placement (put under the role configs).

But the docs aren't very clear towards the end. I can't see the table as per the docs:

I see:

NickLarsenNZ commented 8 months ago

Doc updates in https://github.com/stackabletech/demos/pull/29

NickLarsenNZ commented 8 months ago

We consider this task done, with the changes here: https://github.com/stackabletech/demos/pull/24

stackabletech / issues