search

DataBiosphere / dsub

Open-source command-line tool to run batch computing tasks and workflows on backend services such as Google Cloud.
Apache License 2.0
265 stars 44 forks source link

does dsub work with dataproc API? #267

Closed jqian2015 closed 1 year ago

jqian2015 commented 1 year ago

A quick question, can we call google Cloud Dataproc API when using dsub? thanks.

wnojopra commented 1 year ago

Hi @jqian2015 , I'm not aware of any reason why this wouldn't work. Something like

dsub \
  --provider google-cls-v2 \
  --project aaa-willyn-test \
  --logging gs://willyn-dsub-test/temp/ \
  --image 'gcr.io/google.com/cloudsdktool/cloud-sdk:slim' \
  --command 'gcloud dataproc clusters list --region us-central1' \
  --regions=us-central1

works for me

jqian2015 commented 1 year ago

thanks a lot, very helpful!