`internal_ip_only = false` flag is ignored in `google_dataproc_cluster`

gtupak commented 1 month ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.5.2 on darwin_arm64

provider registry.terraform.io/hashicorp/google v5.34.0
provider registry.terraform.io/hashicorp/google-beta v5.34.0

Affected Resource(s)

google_dataproc_cluster

Terraform Configuration

resource "google_storage_bucket" "dbt_python" {
  name     = "${var.environment}--dbt-python"
  location = var.region
  labels   = local.labels
}

resource "google_dataproc_cluster" "dataproc_cluster" {
  name   = "dataproc-cluster"
  region = var.region
  labels = local.labels

  graceful_decommission_timeout = "3600s"

  cluster_config {
    staging_bucket = google_storage_bucket.dbt_python.id
    temp_bucket    = google_storage_bucket.dbt_python.id

    gce_cluster_config {
      internal_ip_only = false
      metadata = {
        "spark-bigquery-connector-version" = "0.36.3"
        "PIP_PACKAGES"                     = "web3==6.19.0 pydantic==2.7.1"
      }
    }

    software_config {
      # Dataproc v2.2 has python 3.11
      # https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.2
      #
      # There's a bug in terraform where if we set version 2.2, it will set
      # internal_ip_only to true even if we set it to false. This is because starting
      # Feb 16 2024, internal ip only is set to True by default and it seems that
      # terraform doesn't account for this. See the release notes of 16 feb 2024:
      # https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.2
      image_version = "2.2"
      override_properties = {
        "spark:spark.dynamicAllocation.enabled" = "true",
        "dataproc:efm.spark.shuffle"            = "primary-worker"
      }
    }

    master_config {
      num_instances = 1
      machine_type  = "n2-standard-4"
    }

    worker_config {
      num_instances = 2
      machine_type = "c2-standard-4"
    }

    autoscaling_config {
      policy_uri = google_dataproc_autoscaling_policy.autoscaling_policy.id
    }

    initialization_action {
      script      = "gs://${google_storage_bucket.dbt_python.name}/dataproc-initialization-actions/connectors.sh"
      timeout_sec = 300
    }

    initialization_action {
      script      = "gs://${google_storage_bucket.dbt_python.name}/dataproc-initialization-actions/pip_install.sh"
      timeout_sec = 300
    }
  }
}

resource "google_dataproc_autoscaling_policy" "autoscaling_policy" {
  policy_id = "dataproc-autoscaling-policy"
  location  = var.region

  worker_config {
    min_instances = 2
    max_instances = 2
  }

  secondary_worker_config {
    min_instances = 0
    max_instances = 32
  }

  basic_algorithm {
    cooldown_period = "120s"
    yarn_config {
      graceful_decommission_timeout = "0s"
      scale_up_factor               = 0.8
      scale_down_factor             = 0.8
    }
  }
}

Debug Output

https://gist.github.com/gtupak/fccc66841bfbc79bc5b42431abe62318

Expected Behavior

The dataproc cluster should have been initialized with Internal IP Only to false as specified in the terraform config.

Actual Behavior

The internal_ip_only config is ignored and the cluster has a private IP. This results in the inability to install 3rd party PyPI packages in the cluster.

Steps to reproduce

terraform apply

Important Factoids

Starting from Dataproc v2.2, the clusters have private IP only turned on by default. See the release notes

References

No response

b/348680869

ggtisc commented 1 month ago

Hi @gtupak!

After creating a google_dataproc_cluster with a cluster_config.gce_cluster_config.internal_ip_only = false the result was successful without errors, even running a 2nd terraform apply the message was Your infrastructure matches the configuration and checking in console for the created resource on dataproc clusters in the cluster configuration the Internal IP only property value was = No.

I suggest you check the same after running the terraform apply in the console to confirm if the value was set according to the terraform configuration.

gtupak commented 1 month ago

Thank you for the prompt reply! I just tried again and got the same result. Are you sure you are specifying the image_version = "2.2"? If I try the 2.1 version it works fine.

ggtisc commented 1 month ago

Confirmed issue!

With cluster_config.software_config.image_version = "2.2" even if the code specifies the attribute cluster_config.gce_cluster_config.internal_ip_only = false after terraform apply in the Google Cloud console dataproc clusters, the created closter configuration Internal IP only attribute appears with a value = Yes

hashicorp / terraform-provider-google