stackabletech / issues

This repository is only for issues that concern multiple repositories or don't fit into any specific repository
2 stars 0 forks source link

chore(tracking): Check and update getting-started scripts for 24.11 #672

Closed NickLarsenNZ closed 1 day ago

NickLarsenNZ commented 2 days ago

Post-Release Getting Started Script Updates

Part of https://github.com/stackabletech/issues/issues/647

[!TIP] It is handy to have a separate shell open watching the images being pulled into KinD:

watch -cd 'docker exec kind-control-plane crictl images -o yaml \
| yq -r ".images[] | .repoTags[] | select(. | contains(\"stackable\"))" \
| grep --color=always -E "^|24\.11\.0"'

This yq filter is based on the proper yq which is a wrapper around jq and can be installed with pip install yq. The filter would need adapting if using the mikefarah yq.

In each operator repository, run the following commands. If any updates are required, open a PR using the applicable link below.

# switch to the release branch
git fetch
git checkout release-24.11
git pull

# Some of the scripts are in a code/ subdirectory
# pushd docs/modules/superset/examples/getting_started
# pushd docs/modules/superset/examples/getting_started/code
pushd $(fd -td getting_started | grep examples); cd code 2>/dev/null || true

# Make a fresh cluster (~12 seconds)
kind delete cluster && kind create cluster
./getting_started.sh stackablectl

# Make a fresh cluster (~12 seconds)
kind delete cluster && kind create cluster
./getting_started.sh helm

popd

[!TIP]

Use a

git checkout -b fix/getting-started-post-24.11

Replace the items in the task lists below with the applicable Pull Requests (if any).

[!CAUTION] Pull requests specific to the release should be made against the release branch. General fixes should be made against main and cherry-picked into a new PR for the release-branch.

### Getting Started Script Checks and Updates
- [x] airflow-operator
- [x] ~commons-operator~
- [x] druid-operator
- [x] ~edc-operator~
- [x] hbase-operator
- [ ] https://github.com/stackabletech/hdfs-operator/pull/611
- [ ] https://github.com/stackabletech/hdfs-operator/pull/612
- [x] ~hello-world-operator~
- [x] hive-operator
- [x] kafka-operator
- [x] ~listener-operator~
- [x] nifi-operator
- [x] opa-operator
- [x] ~secret-operator~
- [x] spark-k8s-operator
- [x] superset-operator
- [x] trino-operator
- [x] zookeeper-operator
NickLarsenNZ commented 2 days ago

🟢 airflow-operator

Noticed docker.stackable.tech/stackable/tools:1.0.0-stackable24.7.0 being pulled. It appears to come from here:

https://github.com/stackabletech/secret-operator/blob/5e348c67da7d788b9ac3384da5e7889faa28d0ab/deploy/helm/secret-operator/templates/secret_migration_job.yaml#L33

It will be removed in SDP 25.3 (https://github.com/stackabletech/secret-operator/issues/477)

NickLarsenNZ commented 2 days ago

🟢 hdfs-operator

There was an error applying. But when I went to check pods, they were all up and running.

error: timed out waiting for the condition

Will run again -> Same problem.

I'm going to increase the wait timer to 10m (todo: raise a PR to main, cherry pick it to the release branch) -> That worked.

I also noticed the webhdfs StatefulSet uses docker.stackable.tech/stackable/testing-tools:0.2.0-stackable0.0.0-dev.

NickLarsenNZ commented 2 days ago

🟢 hive-operator

The hive-test-helper StatefulSet uses docker.stackable.tech/stackable/testing-tools:0.2.0-stackable0.0.0-dev.

NickLarsenNZ commented 2 days ago

🟢 kafka-operator

Need to add kcat to the shell.nix (templating).

NickLarsenNZ commented 2 days ago

🟢 trino-operator

The image is way too large. I had to docker pull and kind load docker-image it, then run getting_started.

Image