search

ankurchavda / streamify

A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
563 stars 118 forks source link

Error while provisioning spark cluster with terraform #5

Closed topefolorunso closed 1 year ago

topefolorunso commented 2 years ago

I used e2-standard-2 for the kafka and airflow instances I then used e2-medium for both master and worker in the spark cluster Apparently, the kafka and airflow instances were successful.

google_dataproc_cluster.mulitnode_spark_cluster: Creating...
╷
│ Error: Error creating Dataproc cluster: googleapi: Error 400: Multiple validation errors:
│  - Insufficient 'CPUS' quota. Requested 6.0, available 4.0.
│  - Insufficient 'IN_USE_ADDRESSES' quota. Requested 3.0, available 2.0.
│  - This request exceeds CPU quota. Some things to try: request fewer workers (a minimum of 2 is required), 
use smaller master and/or worker machine types (such as n1-standard-2)., badRequest
│
│   with google_dataproc_cluster.mulitnode_spark_cluster,
│   on main.tf line 93, in resource "google_dataproc_cluster" "mulitnode_spark_cluster":
│   93: resource "google_dataproc_cluster" "mulitnode_spark_cluster" {
│
ankurchavda commented 2 years ago

Hi Tope,

Have you availed the 300$ in credit by creating a new GCP account? Can you check this link and see if you find something here?

topefolorunso commented 2 years ago

Thanks. The problem was with the quota available for the free trial. Apparently, I can't increase it.

What I did instead was to use 1 master which acts as worker as well. I hope that works as well.

ankurchavda commented 2 years ago

Glad you got it to work. I guess it should work alright.