search

GoogleCloudPlatform / data-science-on-gcp

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Apache License 2.0
1.31k stars 715 forks source link

Ch7 - can't increase quota of Dataproc cluster; workaround not working #163

Open so-james-gammerman opened 1 year ago

so-james-gammerman commented 1 year ago

Unfortunately I can't run create_large_cluster.sh :

 - Insufficient 'CPUS' quota. Requested 404.0, available 12.0.
 - Insufficient 'CPUS_ALL_REGIONS' quota. Requested 404.0, available 20.0.
 - Insufficient 'DISKS_TOTAL_GB' quota. Requested 25500.0, available 2596.0.
 - Insufficient 'INSTANCES' quota. Requested 51.0, available 21.0.
 - Insufficient 'IN_USE_ADDRESSES' quota. Requested 51.0, available 5.0.

The chapter 7 README says: If it fails with quota issues, get increased quota. If you can't have more quota, reduce the number of workers appropriately

Unfortunately if you're using the GCP Free trial then you can't get the increased quota, and I've tried reducing the number of workers several times but the requested number of CPUs is still too large- this was the output with the following parameters: --num-workers 3 --num-secondary-workers 3:

 - Insufficient 'CPUS' quota. Requested 52.0, available 12.0.
 - Insufficient 'CPUS_ALL_REGIONS' quota. Requested 52.0, available 20.0.
 - Insufficient 'DISKS_TOTAL_GB' quota. Requested 3500.0, available 2596.0.
 - Insufficient 'IN_USE_ADDRESSES' quota. Requested 7.0, available 5.0.

Any advice would be appreciated otherwise I'll have to give up on the rest of this chapter...

lakshmanok commented 1 year ago

Yes, you may not be able to do the large cluster on the free trial. You could do the rest of the chapter, but sticking to the small dataset that the small cluster is sufficient for.

thanks, Lak

On Wed, Feb 1, 2023, 10:27 AM so-james-gammerman @.***> wrote:

Unfortunately I can't run create_large_cluster.sh :

  • Insufficient 'CPUS' quota. Requested 404.0, available 12.0.
  • Insufficient 'CPUS_ALL_REGIONS' quota. Requested 404.0, available 20.0.
  • Insufficient 'DISKS_TOTAL_GB' quota. Requested 25500.0, available 2596.0.
  • Insufficient 'INSTANCES' quota. Requested 51.0, available 21.0.
  • Insufficient 'IN_USE_ADDRESSES' quota. Requested 51.0, available 5.0.

The chapter 7 README says: If it fails with quota issues, get increased quota. If you can't have more quota, reduce the number of workers appropriately

Unfortunately if you're using the GCP Free trial then you can't get the increased quota, and I've tried reducing the number of workers several times but the requested number of CPUs is still too large:

  • Insufficient 'CPUS' quota. Requested 52.0, available 12.0.
  • Insufficient 'CPUS_ALL_REGIONS' quota. Requested 52.0, available 20.0.
  • Insufficient 'DISKS_TOTAL_GB' quota. Requested 3500.0, available 2596.0.
  • Insufficient 'IN_USE_ADDRESSES' quota. Requested 7.0, available 5.0.

Any advice would be appreciated otherwise I'll have to give up on the rest of this chapter...

— Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/data-science-on-gcp/issues/163, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANJPZYRMRTJFLBRAJYUACDWVJ6HLANCNFSM6AAAAAAUN3XS3U . You are receiving this because you are subscribed to this thread.Message ID: @.***>