Closed sbernauer closed 1 year ago
It kept running over the weekend. We are now at
|files|total_size |avg_size |total_records |avg_records|smallest_file|
|-----|---------------|-----------|--------------|-----------|-------------|
|298 |150,318,197,593|504,423,481|36,430,087,583|122,248,616|7,101,325 |
@maltesander could you please do another round reviewing the docs?
Many many thanks!
bors r+
Description
Needs a larger k8s cluster! I use IONOS k8s with 9x 4 cores (8 threads), 20GB ram and 30GB hdd disk Maybe we can also offer a smaller variant later on.
Otherwise business as usual. From feature-branch run
stackablectl --additional-stacks-file stacks/stacks-v1.yaml --additional-releases-file releases.yaml --additional-demos-file demos/demos-v1.yaml demo install data-warehouse-iceberg-trino-spark
I'm not happy with some parts but i think an iterative approach is best:
To get to the Spark UI
kubectl port-forward $(kubectl get pod -o name | grep 'spark-ingest-into-warehouse-.*-driver') 4040
Review Checklist
Once the review is done, comment
bors r+
(orbors merge
) to merge. Further information