search

kubeflow / website

Kubeflow's public website
Creative Commons Attribution 4.0 International
146 stars 767 forks source link

[Feedback] docs/components/spark-operator/user-guide/gcp.md #3843

Open nrawal47 opened 1 week ago

nrawal47 commented 1 week ago

The specified image does not exist.

andreyvelich commented 1 week ago

/assign @ChenYi015 @yuchaoran2011 @vara-bonthu @jacobsalway

jacobsalway commented 1 week ago

This is referring to gcr.io/ynli-k8s/spark:v2.3.0-gcs right?

ChenYi015 commented 1 week ago

The GCP related documentation have not been updated for years, I have no idea whether it is still applicable now.

andreyvelich commented 1 week ago

I would suggest to remove this documentation page if that doesn't work now.

nrawal47 commented 1 week ago

To be clear, this is the jar that works for us:

jars:
        - "local:///opt/spark/jobs/lib/gcs-connector-3.0.2-shaded.jar"

relevant hadoopConf:

fs.AbstractFileSystem.gs.impl: "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS"
fs.gs.impl: "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem" # testing
fs.gs.auth.service.account.enable: "false"
fs.gs.auth.type: "UNAUTHENTICATED"