status-im / infra-eth-cluster

Infrastructure for Status-go fleets
https://github.com/status-im/status-go
0 stars 0 forks source link

Rename eth.beta to eth.prod fleet #23

Closed jakubgs closed 4 years ago

jakubgs commented 5 years ago

Since we are nearing V1 it would make sense to rename our eth.beta fleet eth.prod.

jakubgs commented 5 years ago

I've checked this and it appears I cannot rename Google Cloud External IP addresses:

  # module.whisper.module.gc-us-central1-a.google_compute_address.host[7] must be replaced
-/+ resource "google_compute_address" "host" {
      ~ address            = "35.224.150.136" -> (known after apply)
        address_type       = "EXTERNAL"
      ~ creation_timestamp = "2018-09-08T19:37:58.877-07:00" -> (known after apply)
      ~ id                 = "russia-servers/us-central1/node-08-gc-us-central1-a-eth-beta" -> (known after apply)
      ~ name               = "node-08-gc-us-central1-a-eth-beta" -> "node-08-gc-us-central1-a-eth-prod" # forces replacement
      ~ network_tier       = "PREMIUM" -> (known after apply)
      ~ project            = "russia-servers" -> (known after apply)
      ~ region             = "us-central1" -> (known after apply)
      ~ self_link          = "https://www.googleapis.com/compute/v1/projects/russia-servers/regions/us-central1/addresses/node-08-gc-us-central1-a-eth-beta" -> (known after apply)
      + subnetwork         = (known after apply)
      ~ users              = [
          - "https://www.googleapis.com/compute/v1/projects/russia-servers/zones/us-central1-a/instances/node-08-gc-us-central1-a-eth-beta",
        ] -> (known after apply)

      - timeouts {}
    }

Which causes them to be re-created, which discards the reserved IP address.

jakubgs commented 5 years ago

I've checked the API and there is no mention of ability to rename a GC address: https://cloud.google.com/compute/docs/reference/rest/v1/addresses This Stack Overflow answer seems to confirm this: https://stackoverflow.com/questions/22687968/rename-reserved-ip-address-in-gce

jakubgs commented 5 years ago

The simplest solution would be to remove those addresses from the Terraform state and let them exist while we create new prod ones and use those. But according to the docs:

Only one resource at a time can use a static external IP address. https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address

jakubgs commented 5 years ago

So either I have to replace external IP addresses for Google Cloud instances in the eth.beta fleet when renaming it, or I have to write some kind of absolute dreadful hack into infra-tf-google-cloud Terraform module to let the hosts keep their wrongly named IP addresses.

1376072970686

corpetty commented 5 years ago

after a brief discussion, we think breaking it is fine under the assumption that the client switches efficiently when it tries to hit a dead IP. Relevant work there:

jakubgs commented 5 years ago

I've checked the current stable Beta version and it also switches when a mailserver is dead, it just does it randomly instead of using any kind of ping, so this shouldn't disturb users much.

jakubgs commented 5 years ago

Changes are prepared in the rename-beta-to-prod branch. I'm going to wait until Adam deploys his new status-go changes to eth.beta so I can get the peer metrics for the fleet before proceeding: https://grafana.status.im/d/gxQG_R1Zk/status-peers

jakubgs commented 4 years ago

Instead of renaming I've created a new eth.prod fleet in: 9515e7c4

The old eth.beta fleet will be kept up for a few weeks after V1 is out to let users trickle out.