search

vmware-archive / pcf-pipelines

PCF Pipelines
Apache License 2.0
158 stars 282 forks source link

GCP: create-infrastructure tries to delete SSL cert that is in use #106

Closed mhausmann-pivotal closed 7 years ago

mhausmann-pivotal commented 7 years ago

GCP is throwing this error during apply, when trying to delete(!?) the SSL cert used by HTTPS Load Balancer:

* google_compute_ssl_certificate.ssl-cert (destroy): 1 error(s) occurred:

* google_compute_ssl_certificate.ssl-cert: Error deleting ssl certificate: googleapi: Error 400: The ssl_certificate resource 'projects/<redacted>/global/sslCertificates/<redacted>-gcp-lb-cert' is already being used by 'projects/<redacted>/global/targetHttpsProxies/<redacted>-gcp-https-proxy', resourceInUseByAnotherResource

Deploy fails as a result.

cf-gitbot commented 7 years ago

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

krishicks commented 7 years ago

Which task are you running, and which version of pcf-pipelines?

abbyachau commented 7 years ago

Hi @mhausmann-pivotal, please let us know if you are still running into issues by letting us know which tasks you are running, and which version of pcf-pipelines you are using. Thanks.

jedsalazar commented 7 years ago

I'm having this issue as well.

* google_compute_ssl_certificate.ssl-cert: Error deleting ssl certificate: googleapi: Error 400: The ssl_certificate resource 'projects/<redacted>/global/sslCertificates/<redacted>-lb-cert' is already being used by 'projects/<redacted>/global/targetHttpsProxies/<redacted>-https-proxy', resourceInUseByAnotherResource

This occurs on the infrastructure-create task. Running the latest version of pcf-pipelines.

krishicks commented 7 years ago

It would be helpful if someone would attach the log from Concourse showing what the plan was that led to this happening. I don't believe I've seen it in our own pipeline, and we rebuilt them recently so our history is gone, anyway.

The plan that Terraform makes should be entirely redacted already, so it should be safe to paste. You still might scan through it to make sure there aren't any secrets you don't want divulged.

jedsalazar commented 7 years ago

Kris,

Here's the requested logs


~ google_compute_firewall.cf-ssh-proxy
    source_ranges.#:          "1" => "0"
    source_ranges.1080289494: "0.0.0.0/0" => ""

~ google_compute_firewall.cf-tcp
    source_ranges.#:          "1" => "0"
    source_ranges.1080289494: "0.0.0.0/0" => ""

-/+ google_compute_ssl_certificate.ssl-cert
    certificate: "-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----" (forces new resource)
    description: "user provided ssl private key / ssl certificate pair" => "user provided ssl private key / ssl certificate pair"
    name:        "gcp-redacted-lb-cert" => "gcp-redacted-lb-cert"
    private_key: "-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----" (forces new resource)
    self_link:   "https://www.googleapis.com/compute/v1/projects/ps-redacted/global/sslCertificates/gcp-redacted-lb-cert" => "<computed>"

~ google_compute_target_https_proxy.https_lb_proxy
    ssl_certificates.#: "1" => "<computed>"

-/+ google_sql_database.account
    instance:  "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:      "account" => "account"
    self_link: "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/databases/account" => "<computed>"

-/+ google_sql_database.app_usage_service
    instance:  "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:      "app_usage_service" => "app_usage_service"
    self_link: "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/databases/app_usage_service" => "<computed>"

-/+ google_sql_database.autoscale
    instance:  "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:      "autoscale" => "autoscale"
    self_link: "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/databases/autoscale" => "<computed>"

-/+ google_sql_database.ccdb
    instance:  "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:      "ccdb" => "ccdb"
    self_link: "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/databases/ccdb" => "<computed>"

-/+ google_sql_database.console
    instance:  "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:      "console" => "console"
    self_link: "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/databases/console" => "<computed>"

-/+ google_sql_database.diego
    instance:  "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:      "diego" => "diego"
    self_link: "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/databases/diego" => "<computed>"

-/+ google_sql_database.locket
    instance:  "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:      "locket" => "locket"
    self_link: "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/databases/locket" => "<computed>"

-/+ google_sql_database.networkpolicyserver
    instance:  "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:      "networkpolicyserver" => "networkpolicyserver"
    self_link: "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/databases/networkpolicyserver" => "<computed>"

-/+ google_sql_database.nfsvolume
    instance:  "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:      "nfsvolume" => "nfsvolume"
    self_link: "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/databases/nfsvolume" => "<computed>"

-/+ google_sql_database.notifications
    instance:  "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:      "notifications" => "notifications"
    self_link: "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/databases/notifications" => "<computed>"

-/+ google_sql_database.routing
    instance:  "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:      "routing" => "routing"
    self_link: "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/databases/routing" => "<computed>"

-/+ google_sql_database.silk
    instance:  "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:      "silk" => "silk"
    self_link: "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/databases/silk" => "<computed>"

-/+ google_sql_database.uaa
    instance:  "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:      "uaa" => "uaa"
    self_link: "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/databases/uaa" => "<computed>"

-/+ google_sql_database_instance.master
    database_version:                                          "MYSQL_5_6" => "MYSQL_5_6"
    ip_address.#:                                              "1" => "<computed>"
    name:                                                      "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    region:                                                    "us-west1" => "us-west1"
    self_link:                                                 "https://www.googleapis.com/sql/v1beta4/projects/ps-redacted/instances/gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "<computed>"
    settings.#:                                                "1" => "1"
    settings.0.crash_safe_replication:                         "false" => "<computed>"
    settings.0.disk_autoresize:                                "true" => "true"
    settings.0.ip_configuration.#:                             "1" => "1"
    settings.0.ip_configuration.0.authorized_networks.#:       "5" => "5"
    settings.0.ip_configuration.0.authorized_networks.0.name:  "nat-1" => "nat-1"
    settings.0.ip_configuration.0.authorized_networks.0.value: "x.x.43.130" => "x.x.43.130"
    settings.0.ip_configuration.0.authorized_networks.1.name:  "nat-2" => "nat-2"
    settings.0.ip_configuration.0.authorized_networks.1.value: "x.x.42.134" => "x.x.42.134"
    settings.0.ip_configuration.0.authorized_networks.2.name:  "nat-3" => "nat-3"
    settings.0.ip_configuration.0.authorized_networks.2.value: "x.x.34.118" => "x.x.34.118"
    settings.0.ip_configuration.0.authorized_networks.3.name:  "opsman" => "opsman"
    settings.0.ip_configuration.0.authorized_networks.3.value: "x.x.11.134" => "x.x.11.134"
    settings.0.ip_configuration.0.authorized_networks.4.name:  "all" => "all"
    settings.0.ip_configuration.0.authorized_networks.4.value: "0.0.0.0/0" => "0.0.0.0/0"
    settings.0.ip_configuration.0.ipv4_enabled:                "true" => "true"
    settings.0.tier:                                           "db-f1-micro" => "db-f1-micro"
    settings.0.version:                                        "1" => "<computed>"

-/+ google_sql_user.account
    host:     "%" => "%"
    instance: "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:     "bosh" => "bosh"
    password: "<sensitive>" => "<sensitive>" (attribute changed)

-/+ google_sql_user.app_usage_service
    host:     "%" => "%"
    instance: "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:     "bosh" => "bosh"
    password: "<sensitive>" => "<sensitive>" (attribute changed)

-/+ google_sql_user.autoscale
    host:     "%" => "%"
    instance: "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:     "bosh" => "bosh"
    password: "<sensitive>" => "<sensitive>" (attribute changed)

-/+ google_sql_user.ccdb
    host:     "%" => "%"
    instance: "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:     "bosh" => "bosh"
    password: "<sensitive>" => "<sensitive>" (attribute changed)

-/+ google_sql_user.diego
    host:     "%" => "%"
    instance: "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:     "bosh" => "bosh"
    password: "<sensitive>" => "<sensitive>" (attribute changed)

-/+ google_sql_user.locket
    host:     "%" => "%"
    instance: "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:     "bosh" => "bosh"
    password: "<sensitive>" => "<sensitive>" (attribute changed)

-/+ google_sql_user.network_policy_server
    host:     "%" => "%"
    instance: "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:     "bosh" => "bosh"
    password: "<sensitive>" => "<sensitive>" (attribute changed)

-/+ google_sql_user.nfs_volume
    host:     "%" => "%"
    instance: "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:     "bosh" => "bosh"
    password: "<sensitive>" => "<sensitive>" (attribute changed)

-/+ google_sql_user.notifications
    host:     "%" => "%"
    instance: "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:     "bosh" => "bosh"
    password: "<sensitive>" => "<sensitive>" (attribute changed)

-/+ google_sql_user.routing
    host:     "%" => "%"
    instance: "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:     "bosh" => "bosh"
    password: "<sensitive>" => "<sensitive>" (attribute changed)

-/+ google_sql_user.silk
    host:     "%" => "%"
    instance: "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:     "bosh" => "bosh"
    password: "<sensitive>" => "<sensitive>" (attribute changed)

-/+ google_sql_user.uaa
    host:     "%" => "%"
    instance: "gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950" => "gcp-redacted-sql-9dc2bc5a-77fa-4a56-80f9-040ffec20174" (forces new resource)
    name:     "bosh" => "bosh"
    password: "<sensitive>" => "<sensitive>" (attribute changed)

Plan: 27 to add, 3 to change, 27 to destroy.
google_compute_ssl_certificate.ssl-cert: Destroying... (ID: gcp-redacted-lb-cert)
google_sql_user.silk: Destroying... (ID: gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/bosh)
google_compute_firewall.cf-tcp: Modifying... (ID: gcp-redacted-allow-cf-tcp)
  source_ranges.#:          "1" => "0"
  source_ranges.1080289494: "0.0.0.0/0" => ""
google_compute_firewall.cf-ssh-proxy: Modifying... (ID: gcp-redacted-allow-ssh-proxy)
  source_ranges.#:          "1" => "0"
  source_ranges.1080289494: "0.0.0.0/0" => ""
google_sql_user.silk: Destruction complete
google_sql_user.locket: Destroying... (ID: gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950/bosh)
google_compute_firewall.cf-tcp: Still modifying... (ID: gcp-redacted-allow-cf-tcp, 10s elapsed)
google_compute_firewall.cf-ssh-proxy: Still modifying... (ID: gcp-redacted-allow-ssh-proxy, 10s elapsed)
google_compute_firewall.cf-tcp: Modifications complete (ID: gcp-redacted-allow-cf-tcp)
google_compute_firewall.cf-ssh-proxy: Modifications complete (ID: gcp-redacted-allow-ssh-proxy)
Error applying plan:

2 error(s) occurred:

* google_sql_user.locket (destroy): 1 error(s) occurred:

* google_sql_user.locket: Error, failed to deleteuser bosh in instance gcp-redacted-sql-88dd9b7b-6b3d-4e1a-8d75-b4dcb5b2d950: googleapi: Error 503: Service temporarily unavailable., serverException
* google_compute_ssl_certificate.ssl-cert (destroy): 1 error(s) occurred:

* google_compute_ssl_certificate.ssl-cert: Error deleting ssl certificate: googleapi: Error 400: The ssl_certificate resource 'projects/ps-redacted/global/sslCertificates/gcp-redacted-lb-cert' is already being used by 'projects/ps-redacted/global/targetHttpsProxies/gcp-redacted-https-proxy', resourceInUseByAnotherResource

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

It looks like the pipeline is creating new certs (because they're tainted?) and it has a problem deleting the old certs because they're already in use by the load balancer. Any ideas why terraform considers the existing ssl certs tainted if that's indeed what's happening?

krishicks commented 7 years ago

Terraform is correctly showing that the cert and private key have changed. The reason is because of this: https://github.com/pivotal-cf/pcf-pipelines/blob/master/install-pcf/gcp/tasks/create-infrastructure/task.sh#L17

If you re-run create-infrastructure without having set the cert and key (meaning changed it from 'generate'), it will make a new one for you.

It's a Terraform bug that it fails to delete the old cert.

krishicks commented 7 years ago

Also, the way around it for now is, after you've initially run create-infrastructure, update your params to supply the generated certs so they aren't recreated.

phopper-pivotal commented 7 years ago

KI is not resolving issue here. after 1st run of create-infrastruture the cert/key were created. hit error with SSL (see below). took cert/key and placed in params.yml and ran set-pipelines. upon suggestion, the VM instances (4) were also deleted but recreated which was good. but on 2nd run of create-infrastructure hit the same error.

-/+ google_compute_ssl_certificate.ssl-cert certificate: "-----BEGIN CERTIFICATE-----\nMIIDcT*\n-----END CERTIFICATE-----" => "-----BEGIN CERTIFICATE-----\nMIIDcTC****-----END CERTIFICATE-----\n" (forces new resource) description: "user provided ssl private key / ssl certificate pair" => "user provided ssl private key / ssl certificate pair" name: "lol-concourse-terraform-lb-cert" => "lol-concourse-terraform-lb-cert" private_key: "-----BEGIN RSA PRIVATE KEY-----\nMIIEow*Ubg==\n-----END RSA PRIVATE KEY-----\n" (forces new resource) self_link: "https://www.googleapis.com/compute/v1/projects/fe-phopper/global/sslCertificates/lol-concourse-terraform-lb-cert" => ""

~ google_compute_target_https_proxy.https_lb_proxy ssl_certificates.#: "1" => ""

~ google_sql_database_instance.master settings.0.ip_configuration.0.authorized_networks.0.value: "104.197.14.103" => "${google_compute_instance.nat-gateway-pri.network_interface.0.access_config.0.assigned_nat_ip}" settings.0.ip_configuration.0.authorized_networks.1.value: "104.197.136.89" => "${google_compute_instance.nat-gateway-sec.network_interface.0.access_config.0.assigned_nat_ip}" settings.0.ip_configuration.0.authorized_networks.2.value: "104.197.17.142" => "${google_compute_instance.nat-gateway-ter.network_interface.0.access_config.0.assigned_nat_ip}" settings.0.ip_configuration.0.authorized_networks.3.value: "130.211.178.85" => "${google_compute_instance.ops-manager.network_interface.0.access_config.0.assigned_nat_ip}"

Plan: 5 to add, 4 to change, 1 to destroy. google_compute_ssl_certificate.ssl-cert: Destroying... (ID: lol-concourse-terraform-lb-cert) google_compute_instance.ops-manager: Creating... can_ip_forward: "" => "false" create_timeout: "" => "4" disk.#: "" => "1" disk.0.auto_delete: "" => "true" disk.0.disk_encryption_key_sha256: "" => "" disk.0.image: "" => "opsman-pcf-gcp-1-11-10" disk.0.size: "" => "50" machine_type: "" => "n1-standard-2" metadata_fingerprint: "" => "" name: "" => "lol-concourse-terraform-ops-manager" network_interface.#: "" => "1" network_interface.0.access_config.#: "" => "1" network_interface.0.access_config.0.assigned_nat_ip: "" => "" network_interface.0.access_config.0.nat_ip: "" => "130.211.178.85" network_interface.0.address: "" => "" network_interface.0.name: "" => "" network_interface.0.subnetwork: "" => "lol-concourse-terraform-subnet-infrastructure-us-central1" self_link: "" => "" tags.#: "" => "2" tags.3904091835: "" => "allow-https" tags.4161136539: "" => "lol-concourse-terraform-opsman" tags_fingerprint: "" => "" zone: "" => "us-central1-a" google_compute_instance.nat-gateway-pri: Creating... can_ip_forward: "" => "true" create_timeout: "" => "4" disk.#: "" => "1" disk.0.auto_delete: "" => "true" disk.0.disk_encryption_key_sha256: "" => "" disk.0.image: "" => "ubuntu-1404-trusty-v20160610" machine_type: "" => "n1-standard-4" metadata_fingerprint: "" => "" metadata_startup_script: "" => "#! /bin/bash\nsudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'\nsudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE\n" name: "" => "lol-concourse-terraform-nat-gateway-pri" network_interface.#: "" => "1" network_interface.0.access_config.#: "" => "1" network_interface.0.access_config.0.assigned_nat_ip: "" => "" network_interface.0.address: "" => "" network_interface.0.name: "" => "" network_interface.0.subnetwork: "" => "lol-concourse-terraform-subnet-infrastructure-us-central1" self_link: "" => "" tags.#: "" => "2" tags.4174854278: "" => "lol-concourse-terraform-nat-instance" tags.593552258: "" => "nat-traverse" tags_fingerprint: "" => "" zone: "" => "us-central1-a" google_compute_instance.nat-gateway-ter: Creating... can_ip_forward: "" => "true" create_timeout: "" => "4" disk.#: "" => "1" disk.0.auto_delete: "" => "true" disk.0.disk_encryption_key_sha256: "" => "" disk.0.image: "" => "ubuntu-1404-trusty-v20160610" machine_type: "" => "n1-standard-4" metadata_fingerprint: "" => "" metadata_startup_script: "" => " #! /bin/bash\n sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'\n sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE\n " name: "" => "lol-concourse-terraform-nat-gateway-ter" network_interface.#: "" => "1" network_interface.0.access_config.#: "" => "1" network_interface.0.access_config.0.assigned_nat_ip: "" => "" network_interface.0.address: "" => "" network_interface.0.name: "" => "" network_interface.0.subnetwork: "" => "lol-concourse-terraform-subnet-infrastructure-us-central1" self_link: "" => "" tags.#: "" => "2" tags.4174854278: "" => "lol-concourse-terraform-nat-instance" tags.593552258: "" => "nat-traverse" tags_fingerprint: "" => "" zone: "" => "us-central1-c" google_compute_firewall.cf-tcp: Modifying... (ID: lol-concourse-terraform-allow-cf-tcp) source_ranges.#: "1" => "0" source_ranges.1080289494: "0.0.0.0/0" => "" google_compute_instance.nat-gateway-sec: Creating... can_ip_forward: "" => "true" create_timeout: "" => "4" disk.#: "" => "1" disk.0.auto_delete: "" => "true" disk.0.disk_encryption_key_sha256: "" => "" disk.0.image: "" => "ubuntu-1404-trusty-v20160610" machine_type: "" => "n1-standard-4" metadata_fingerprint: "" => "" metadata_startup_script: "" => " #! /bin/bash\n sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'\n sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE\n " name: "" => "lol-concourse-terraform-nat-gateway-sec" network_interface.#: "" => "1" network_interface.0.access_config.#: "" => "1" network_interface.0.access_config.0.assigned_nat_ip: "" => "" network_interface.0.address: "" => "" network_interface.0.name: "" => "" network_interface.0.subnetwork: "" => "lol-concourse-terraform-subnet-infrastructure-us-central1" self_link: "" => "" tags.#: "" => "2" tags.4174854278: "" => "lol-concourse-terraform-nat-instance" tags.593552258: "" => "nat-traverse" tags_fingerprint: "" => "" zone: "" => "us-central1-b" google_compute_instance.ops-manager: Still creating... (10s elapsed) google_compute_instance.nat-gateway-ter: Still creating... (10s elapsed) google_compute_firewall.cf-tcp: Still modifying... (ID: lol-concourse-terraform-allow-cf-tcp, 10s elapsed) google_compute_instance.nat-gateway-pri: Still creating... (10s elapsed) google_compute_instance.nat-gateway-sec: Still creating... (10s elapsed) google_compute_firewall.cf-tcp: Modifications complete (ID: lol-concourse-terraform-allow-cf-tcp) google_compute_firewall.cf-ssh-proxy: Modifying... (ID: lol-concourse-terraform-allow-ssh-proxy) source_ranges.#: "1" => "0" source_ranges.1080289494: "0.0.0.0/0" => "" google_compute_instance.nat-gateway-ter: Creation complete (ID: lol-concourse-terraform-nat-gateway-ter) google_compute_instance.nat-gateway-pri: Creation complete (ID: lol-concourse-terraform-nat-gateway-pri) google_compute_instance.ops-manager: Creation complete (ID: lol-concourse-terraform-ops-manager) google_compute_instance.nat-gateway-sec: Creation complete (ID: lol-concourse-terraform-nat-gateway-sec) google_sql_database_instance.master: Modifying... (ID: lol-concourse-terraform-pelican) settings.0.ip_configuration.0.authorized_networks.0.value: "104.197.14.103" => "104.197.136.89" settings.0.ip_configuration.0.authorized_networks.1.value: "104.197.136.89" => "104.197.17.142" settings.0.ip_configuration.0.authorized_networks.2.value: "104.197.17.142" => "104.197.247.42" google_compute_firewall.cf-ssh-proxy: Still modifying... (ID: lol-concourse-terraform-allow-ssh-proxy, 10s elapsed) google_compute_firewall.cf-ssh-proxy: Modifications complete (ID: lol-concourse-terraform-allow-ssh-proxy) google_sql_database_instance.master: Still modifying... (ID: lol-concourse-terraform-pelican, 10s elapsed) google_sql_database_instance.master: Still modifying... (ID: lol-concourse-terraform-pelican, 20s elapsed) google_sql_database_instance.master: Still modifying... (ID: lol-concourse-terraform-pelican, 30s elapsed) google_sql_database_instance.master: Modifications complete (ID: lol-concourse-terraform-pelican) Error applying plan:

1 error(s) occurred: