search

GoogleCloudPlatform / cloudml-samples

Cloud ML Engine repo. Please visit the new Vertex AI samples repo at https://github.com/GoogleCloudPlatform/vertex-ai-samples
https://cloud.google.com/ai-platform/docs/
Apache License 2.0
1.52k stars 859 forks source link

Submitting Training job from census tf-keras examples very slow? #437

Closed datistiquo closed 5 years ago

datistiquo commented 5 years ago

Hey,

submitting training job to ai platform from the example 

! gcloud ai-platform jobs submit training $JOB_NAME \
  --package-path trainer/ \
  --module-name trainer.task \
  --region $REGION \
  --python-version 3.5 \
  --runtime-version 1.13 \
  --job-dir $JOB_DIR \
  --stream-logs

in https://github.com/GoogleCloudPlatform/cloudml-samples/blob/master/notebooks/tensorflow/getting-started-keras.ipynb

is very slow before any output log is shown?

Is this due to my machine?

anifort commented 5 years ago

Where are your data?

datistiquo commented 5 years ago

I am using the example from tutorial in above link. So ti will be downloaded.

datistiquo commented 5 years ago

It runs now for 2 hours without any log...

nnegrey commented 5 years ago

You can check the status of the job here: https://console.cloud.google.com/mlengine/jobs/my_first_keras_job/charts/cpu?project=YOUR_PROJECT_ID

datistiquo commented 5 years ago

I'll try oit again. Does the submitting job also depends on the chosen region?

nnegrey commented 5 years ago

Yes it does, please see here for more info: https://cloud.google.com/ml-engine/docs/tensorflow/regions