search

GoogleCloudDataproc / initialization-actions

Run in all nodes of your cluster before the cluster starts - lets you customize your cluster
https://cloud.google.com/dataproc/init-actions
Apache License 2.0
588 stars 512 forks source link

[Oozie] Updated oozie.sh to include rocky8 and rocky9 image versions #1178

Closed TheAkertu closed 4 months ago

cjac commented 7 months ago

/gcbrun

kuldeepkk-dev commented 6 months ago

Tests are failing on 2.2-debian12 and 2.2-ubuntu22. Please check what's happening on these two images.

cjac commented 4 months ago

2.2-debian12 with 3 masters comes up fine

Fri Jul  5 04:41:19 PM PDT 2024
+ gcloud dataproc clusters create cluster-1668020639 --master-boot-disk-type pd-ssd --worker-boot-disk-type pd-ssd --secondary-worker-boot-disk-type pd-ssd --master-boot-disk-size 100 --worker-boot-disk-size 100 --secondary-worker-boot-disk-size 100 --master-machine-type n1-standard-8 --worker-machine-type n1-standard-8 --num-masters=3 --num-workers=3 --num-secondary-workers=0 --region us-central1 --zone us-central1-a --subnet subnet-cluster-1668020639 --no-address --image-version 2.2 --service-account=sa-cluster-1668020639@cjac-2021-00.iam.gserviceaccount.com --tags=tag-cluster-1668020639 --bucket cjac-docker-on-yarn --enable-component-gateway --initialization-action-timeout=35m --max-idle=2d --no-shielded-secure-boot --metadata include-gpus=true --metadata gpu-driver-provider=NVIDIA --initialization-action-timeout=15m --initialization-actions gs://cjac-docker-on-yarn/dataproc-initialization-actions/oozie/oozie.sh --properties spark:spark.executor.resource.gpu.amount=1,spark:spark.task.resource.gpu.amount=1 --metadata init-actions-repo=gs://cjac-docker-on-yarn/dataproc-initialization-actions --metadata install-gpu-agent=true --scopes https://www.googleapis.com/auth/cloud-platform
Waiting on operation [projects/cjac-2021-00/regions/us-central1/operations/df50baa3-c563-3466-be85-c153cc5fde89].
Waiting for cluster creation operation...
WARNING: Consider using Auto Zone rather than selecting a zone manually. See https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/auto-zone
Waiting for cluster creation operation...done.
Created [https://dataproc.googleapis.com/v1/projects/cjac-2021-00/regions/us-central1/clusters/cluster-1668020639] Cluster placed in zone [us-central1-a].

real    5m38.775s
user    0m2.884s
sys     0m0.175s
+ date
Fri Jul  5 04:46:58 PM PDT 2024
cjac commented 4 months ago

2.2-ubuntu22 also succeeds:

Fri Jul  5 04:49:55 PM PDT 2024
+ gcloud dataproc clusters create cluster-1668020639 --master-boot-disk-type pd-ssd --worker-boot-disk-type pd-ssd --secondary-worker-boot-disk-type pd-ssd --master-boot-disk-size 100 --worker-boot-disk-size 100 --secondary-worker-boot-disk-size 100 --master-machine-type n1-standard-8 --worker-machine-type n1-standard-8 --num-masters=3 --num-workers=3 --num-secondary-workers=0 --region us-central1 --zone us-central1-a --subnet subnet-cluster-1668020639 --no-address --image-version 2.2-ubuntu22 --service-account=sa-cluster-1668020639@cjac-2021-00.iam.gserviceaccount.com --tags=tag-cluster-1668020639 --bucket cjac-docker-on-yarn --enable-component-gateway --initialization-action-timeout=35m --max-idle=2d --no-shielded-secure-boot --metadata include-gpus=true --metadata gpu-driver-provider=NVIDIA --initialization-action-timeout=15m --initialization-actions gs://cjac-docker-on-yarn/dataproc-initialization-actions/oozie/oozie.sh --properties spark:spark.executor.resource.gpu.amount=1,spark:spark.task.resource.gpu.amount=1 --metadata init-actions-repo=gs://cjac-docker-on-yarn/dataproc-initialization-actions --metadata install-gpu-agent=true --scopes https://www.googleapis.com/auth/cloud-platform
Waiting on operation [projects/cjac-2021-00/regions/us-central1/operations/3d6a7259-c6ae-3a61-a3e8-42cbf24f9364].
Waiting for cluster creation operation...⠛
WARNING: Consider using Auto Zone rather than selecting a zone manually. See https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/auto-zone
Waiting for cluster creation operation...done.
Created [https://dataproc.googleapis.com/v1/projects/cjac-2021-00/regions/us-central1/clusters/cluster-1668020639] Cluster placed in zone [us-central1-a].

real    5m38.564s
user    0m2.792s
sys     0m0.238s
+ date
Fri Jul  5 04:55:33 PM PDT 2024