search

k8ssandra / k8ssandra-terraform

Apache License 2.0
17 stars 16 forks source link

Hashicorp Terraform 1.0.x bug generating invalid gcp_medusa_key.json #53

Closed johnsmartco closed 3 years ago

johnsmartco commented 3 years ago

Bug Report

Describe the bug Following the instructions in our Install K8ssandra on GKE topic, we noticed that the Medusa container in C* pods would not start. Looking further, it appears that Hashicorp's terraform generates extraneous new-line characters in the private key of gcp_medusa_key.json.

This was a bug in Terraform 0.13 (https://github.com/hashicorp/terraform/issues/25986), supposedly fixed, but appears to be an issue in Terraform 1.0.0. johnsmartco filed a new bug report with Hashicorp, https://github.com/hashicorp/terraform/issues/29079 .

To Reproduce

Expected behavior Hashicorp terraform sw, such as this command, should generate a valid key for Medusa:

terraform output -json service_account_key > medusa_gcp_key.json

Environment (please complete the following information):

NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION prod-k8ssandra default 1 2021-07-06 18:41:56.898509262 +0000 UTC deployed k8ssandra-1.2.0

From my version of gke.values.yaml:

USER-SUPPLIED VALUES: cassandra: cassandraLibDirVolume: size: 2048Gi storageClass: standard-rwo datacenters:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.11", GitCommit:"c6a2f08fc4378c5381dd948d9ad9d1080e3e6b33", GitTreeState:"clean", BuildDate:"2021-05-12T12:27:07Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.10-gke.1600", GitCommit:"7b8e568a7fb4c9d199c2ba29a5f7d76f6b4341c2", GitTreeState:"clean", BuildDate:"2021-05-07T09:18:53Z", GoVersion:"go1.15.10b5", Compiler:"gc", Platform:"linux/amd64"}

Additional context It's a bug in Hashicorp but entering this issue so we can track it. cc: jdonenine.

┆Issue is synchronized with this Jiraserver Bug by Unito ┆Issue Number: K8SSAND-666 ┆Priority: Medium

johnsmartco commented 3 years ago

cc: @adejanovski , @jsanda , @bradfordcp , @jdonenine

So guys, I tried using the terraform output -raw service_account_key > medusa_gcp_key command (instead of the same but with -json as we currently document) to avoid the (double backslashes + n) characters in a medusa_gcp_key.json file. Unfortunately the medusa container does NOT like it:

terraform output -raw service_account_key > medusa_gcp_key

cat medusa_gcp_key { "type": "service_account", "project_id": "gcp-techpubs", "private_key_id": "08efecabc40f93f22014f036f33b1a6c4479ee7f", "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCmMKDLDqK0DEXL\nUyV3b1WwulvtpFKZtPE7uifY21OPdc7ctYkibq1MUB39+/es/NC99QRD30zU9q8Q\nm8k7JlRE8MIYgIN2h1J6vygMuy7O/fxGHK7v0AUyoAHJTl2laa+eqPMraWZF26GP\nkwoxrLU96v0ZroxcJXBNQUV0C8dskH87/lzRETs4J66SSHqGRaspvFoWoJsGT3aj\nno1i5Ruqu5+rscICKCEeyscXvcQXOVgdHZ/HQRb/jTPeyyVg7FXexyTogOfNjlEq\ngjk3oHSpONPQnVIaDoJdPd/0oWmh5EML4XNaJVQyI+iUMXJayh3c4oWwGLI0Ul1e\netkL88cfAgMBAAECggEAECgvIb6xA8cjq/JP9K4eyxk7Ib31h9sHDvL4t8UO+SYP\nb/a4N+BBuecaG1y5RhioJKei/AREIHpBJ+C1bio6TB+KBz7LSSU72BIojyBNmzMn\nKRvVyBvVvYe/1LXpNdEyET9EsOZE2qrRKLe3aH/tePEVHT+Shsze/6vGWFNi1oOH\nmG9BduU7inRUutPZxbAvyInOFu2rBdui5JrpIh7gE1K9fEkrykwfzJipumHVcwO6\nfr5sWVlWZiQYn8CpjOgOsMnQwCx3HCPGeHMDRq7wwcWZpidaHgOPUSMUc4JGTpZI\nr1YT1X+lWb9dMrb6B88v3Zu9rrRjR9uVzCHHFff3FQKBgQDfD4QO2JkYIGi8EISl\n5NeLluhq5JPipUlcdb0RusrMKaPv12kVR7W0ag/aazpTnGcU4bYAYNPKohQ0vNKI\nLmiWCA5TqkOM2h9f9x8UdEZRlCbfcfAt1t9kizS/TIUqdg2+uhuXPwe07jRLac3w\n/Zg6BdDWCdTaqI1xruMNlM0g6wKBgQC+uzMC2Zqc988NODGyIqrVtzTSGLoDjX7Z\nlPe7ijWVjFmbqIr3hcvCMqcF3T97xrEsxezFmW01Pjaw1H6jhdJBTStOPXzYvzST\n77oFfvqdMpZY6EoCZaFfPnFJsiVf8VVzhRIh9DQHJvp8CO7GlLF9wPJBKN0gCfoC\nx7KBSQYFnQKBgEhxHY5EwdOYEuug4bOO0oG/DNRKT8z3qQ+iHTk41MvhaiBmcPY8\nR8LD/0oMMPEehku90gHlcGMKqE1IVYdsqwKusxhw32dLH41nEGzwJjYwSz+eHUaF\n1h8tJoBKNaVImcYrYvKHwdT813b2ca+mbF3LrbsVjk9FFkqpZYvvm37pAoGAQCg+\nL349GQHunvbBGLe6mWsoFGgG4j0NZ2lJr8b84VI1rjuSi1JxSOC6+CkFOsCKRpJJ\nY+qJJrjHstG3y4+vOxIXOBlMzqf6qqyf+UtOR+SFWGDCzG2dS6l4eB65nb1TTil8\nAPnhKmq9JU3n69qGndNyaeKyBx30AE1mSUDkCq0CgYEAmjJKAmhELRoQ0hF34qTM\nLOoyiV/X24J1vfLonRR7N84OyXcPhWMUbvJ4yjSbkm6oScSNSxLyqKMgM1k4U0XF\nKCRwz0tzUiMIAhWU+hbtMoGSdsKnUCPClDen4lnSwp+5ytDGnh2QOQE6MRm7M5ni\nJKHa/PAjLumD1Or/EqftPZ8=\n-----END PRIVATE KEY-----\n", "client_email": "prod-k8ssandra-sa@gcp-techpubs.iam.gserviceaccount.com", "client_id": "114700624897800231906", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/prod-k8ssandra-sa%40gcp-techpubs.iam.gserviceaccount.com" }

kubectl create secret generic prod-k8ssandra-medusa-key --from-file=medusa_gcp_key=./medusa_gcp_key secret/prod-k8ssandra-medusa-key created

grep medusa gke.values.yaml

medusa: storageSecret: prod-k8ssandra-medusa-key

helm install prod-k8ssandra k8ssandra/k8ssandra -f gke.values.yaml ... NAME: prod-k8ssandra LAST DEPLOYED: Thu Jul 8 18:30:55 2021 NAMESPACE: default STATUS: deployed REVISION: 1

k get pods

NAME READY STATUS RESTARTS AGE prod-k8ssandra-cass-operator-56446cc654-jcqrp 1/1 Running 0 2m19s prod-k8ssandra-dc1-stargate-68cb9d56d6-cqgxl 0/1 Init:0/1 0 2m19s prod-k8ssandra-dc1-stargate-68cb9d56d6-f5lzb 0/1 Init:0/1 0 2m19s prod-k8ssandra-dc1-stargate-68cb9d56d6-kcx74 0/1 Init:0/1 0 2m19s prod-k8ssandra-dc1-us-central1-a-sts-0 1/3 Error 4 2m prod-k8ssandra-dc1-us-central1-c-sts-0 1/3 CrashLoopBackOff 2 2m prod-k8ssandra-dc1-us-central1-f-sts-0 2/3 CrashLoopBackOff 4 2m prod-k8ssandra-grafana-5f8d54d5fc-7rnx7 2/2 Running 0 2m20s prod-k8ssandra-kube-promet-operator-fc975b8f4-mwngm 1/1 Running 0 2m19s prod-k8ssandra-medusa-operator-b9577db9-qptpz 1/1 Running 0 2m20s prod-k8ssandra-reaper-operator-d9599c75f-5mncx 1/1 Running 0 2m19s prometheus-prod-k8ssandra-kube-promet-prometheus-0 2/2 Running 1 2m17s

k logs prod-k8ssandra-dc1-us-central1-a-sts-0 -c medusa MEDUSA_MODE = GRPC sleeping for 0 sec Starting Medusa gRPC service /home/cassandra/.local/lib/python3.6/site-packages/requests/init.py:91: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) INFO:root:Init service [2021-07-08 18:34:55,957] INFO: Init service DEBUG:root:Loading storage_provider: google_storage [2021-07-08 18:34:55,957] DEBUG: Loading storage_provider: google_storage Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/cassandra/medusa/service/grpc/server.py", line 158, in medusa_pb2_grpc.add_MedusaServicer_to_server(MedusaService(config), server) File "/home/cassandra/medusa/service/grpc/server.py", line 44, in init self.storage = Storage(config=self.config.storage) File "/home/cassandra/medusa/storage/init.py", line 72, in init self.storage_driver = self._connect_storage() File "/home/cassandra/medusa/storage/init.py", line 78, in _connect_storage google_storage = GoogleStorage(self._config) File "/home/cassandra/medusa/storage/abstract_storage.py", line 39, in init self.driver = self.connect_storage() File "/home/cassandra/medusa/storage/google_storage.py", line 39, in connect_storage with io.open(os.path.expanduser(self.config.key_file), 'r', encoding='utf-8') as json_fi: FileNotFoundError: [Errno 2] No such file or directory: '/etc/medusa-secrets/medusa_gcp_key.json'

:-(

johnsmartco commented 3 years ago

cc: @adejanovski , @jsanda , @bradfordcp , @jdonenine

Ok, success. I finally got all pods/containers including medusa within c* pods to start. Backing up, after defining the env variables, and submitting terraform init, terraform plan, and terraform apply commands, I did the following. Notice especially the order of arguments in the 2nd command below:

terraform output -raw service_account_key > medusa_gcp_key

kubectl create secret generic prod-k8ssandra-medusa-key --from-file=medusa_gcp_key.json=medusa_gcp_key

helm install prod-k8ssandra k8ssandra/k8ssandra -f gke.values.yaml

k get pods NAME READY STATUS RESTARTS AGE prod-k8ssandra-cass-operator-56446cc654-zh4tg 1/1 Running 0 9m52s prod-k8ssandra-dc1-stargate-68cb9d56d6-g2xfp 1/1 Running 4 9m52s prod-k8ssandra-dc1-stargate-68cb9d56d6-kv6dp 1/1 Running 3 9m52s prod-k8ssandra-dc1-stargate-68cb9d56d6-w8bmh 1/1 Running 5 9m52s prod-k8ssandra-dc1-us-central1-a-sts-0 3/3 Running 0 9m37s prod-k8ssandra-dc1-us-central1-c-sts-0 3/3 Running 0 9m37s prod-k8ssandra-dc1-us-central1-f-sts-0 3/3 Running 0 9m37s prod-k8ssandra-grafana-5f8d54d5fc-jv2p6 2/2 Running 0 9m52s prod-k8ssandra-kube-promet-operator-fc975b8f4-rdsbd 1/1 Running 0 9m51s prod-k8ssandra-medusa-operator-b9577db9-wxq4c 1/1 Running 0 9m52s prod-k8ssandra-reaper-747c48d7c6-p9zn9 1/1 Running 0 5m41s prod-k8ssandra-reaper-operator-d9599c75f-l8ccr 1/1 Running 0 9m52s prometheus-prod-k8ssandra-kube-promet-prometheus-0 2/2 Running 1 9m49s

Looks good:

kubectl get secret prod-k8ssandra-medusa-key -o yaml apiVersion: v1 data: medusa_gcp_key.json: ewogICJ0eXBlIjogInNlcnZpY2VfYWNjb3VudCIsCiAgInByb2plY3RfaWQiOiAiZ2NwLXRlY2hwdWJzIiwKICAicHJpdmF0ZV9rZXlfaWQiOiAiMDhlZmVjYWJjNDBmOTNmMjIwMTRmMDM2ZjMzYjFhNmM0NDc5ZWU3ZiIsCiAgInByaXZhdGVfa2V5IjogIi0tLS0tQkVHSU4gUFJJVkFURSBLRVktLS0tLVxuTUlJRXZRSUJBREFOQmdrcWhraUc5dzBCQVFFRkFBU0NCS2N3Z2dTakFnRUFBb0lCQVFDbU1LRExEcUswREVYTFxuVXlWM2IxV3d1bHZ0cEZLWnRQRTd1aWZZMjFPUGRjN2N0WWtpYnExTVVCMzkrL2VzL05DOTlRUkQzMHpVOXE4UVxubThrN0psUkU4TUlZZ0lOMmgxSjZ2eWdNdXk3Ty9meEdISzd2MEFVeW9BSEpUbDJsYWErZXFQTXJhV1pGMjZHUFxua3dveHJMVTk2djBacm94Y0pYQk5RVVYwQzhkc2tIODcvbHpSRVRzNEo2NlNTSHFHUmFzcHZGb1dvSnNHVDNhalxubm8xaTVSdXF1NStyc2NJQ0tDRWV5c2NYdmNRWE9WZ2RIWi9IUVJiL2pUUGV5eVZnN0ZYZXh5VG9nT2ZOamxFcVxuZ2prM29IU3BPTlBRblZJYURvSmRQZC8wb1dtaDVFTUw0WE5hSlZReUkraVVNWEpheWgzYzRvV3dHTEkwVWwxZVxuZXRrTDg4Y2ZBZ01CQUFFQ2dnRUFFQ2d2SWI2eEE4Y2pxL0pQOUs0ZXl4azdJYjMxaDlzSER2TDR0OFVPK1NZUFxuYi9hNE4rQkJ1ZWNhRzF5NVJoaW9KS2VpL0FSRUlIcEJKK0MxYmlvNlRCK0tCejdMU1NVNzJCSW9qeUJObXpNblxuS1J2VnlCdlZ2WWUvMUxYcE5kRXlFVDlFc09aRTJxclJLTGUzYUgvdGVQRVZIVCtTaHN6ZS82dkdXRk5pMW9PSFxubUc5QmR1VTdpblJVdXRQWnhiQXZ5SW5PRnUyckJkdWk1SnJwSWg3Z0UxSzlmRWtyeWt3ZnpKaXB1bUhWY3dPNlxuZnI1c1dWbFdaaVFZbjhDcGpPZ09zTW5Rd0N4M0hDUEdlSE1EUnE3d3djV1pwaWRhSGdPUFVTTVVjNEpHVHBaSVxucjFZVDFYK2xXYjlkTXJiNkI4OHYzWnU5cnJSalI5dVZ6Q0hIRmZmM0ZRS0JnUURmRDRRTzJKa1lJR2k4RUlTbFxuNU5lTGx1aHE1SlBpcFVsY2RiMFJ1c3JNS2FQdjEya1ZSN1cwYWcvYWF6cFRuR2NVNGJZQVlOUEtvaFEwdk5LSVxuTG1pV0NBNVRxa09NMmg5Zjl4OFVkRVpSbENiZmNmQXQxdDlraXpTL1RJVXFkZzIrdWh1WFB3ZTA3alJMYWMzd1xuL1pnNkJkRFdDZFRhcUkxeHJ1TU5sTTBnNndLQmdRQyt1ek1DMlpxYzk4OE5PREd5SXFyVnR6VFNHTG9Ealg3WlxubFBlN2lqV1ZqRm1icUlyM2hjdkNNcWNGM1Q5N3hyRXN4ZXpGbVcwMVBqYXcxSDZqaGRKQlRTdE9QWHpZdnpTVFxuNzdvRmZ2cWRNcFpZNkVvQ1phRmZQbkZKc2lWZjhWVnpoUkloOURRSEp2cDhDTzdHbExGOXdQSkJLTjBnQ2ZvQ1xueDdLQlNRWUZuUUtCZ0VoeEhZNUV3ZE9ZRXV1ZzRiT08wb0cvRE5SS1Q4ejNxUStpSFRrNDFNdmhhaUJtY1BZOFxuUjhMRC8wb01NUEVlaGt1OTBnSGxjR01LcUUxSVZZZHNxd0t1c3hodzMyZExINDFuRUd6d0pqWXdTeitlSFVhRlxuMWg4dEpvQktOYVZJbWNZcll2S0h3ZFQ4MTNiMmNhK21iRjNMcmJzVmprOUZGa3FwWll2dm0zN3BBb0dBUUNnK1xuTDM0OUdRSHVudmJCR0xlNm1Xc29GR2dHNGowTloybEpyOGI4NFZJMXJqdVNpMUp4U09DNitDa0ZPc0NLUnBKSlxuWStxSkpyakhzdEczeTQrdk94SVhPQmxNenFmNnFxeWYrVXRPUitTRldHREN6RzJkUzZsNGVCNjVuYjFUVGlsOFxuQVBuaEttcTlKVTNuNjlxR25kTnlhZUt5QngzMEFFMW1TVURrQ3EwQ2dZRUFtakpLQW1oRUxSb1EwaEYzNHFUTVxuTE9veWlWL1gyNEoxdmZMb25SUjdOODRPeVhjUGhXTVVidko0eWpTYmttNm9TY1NOU3hMeXFLTWdNMWs0VTBYRlxuS0NSd3owdHpVaU1JQWhXVStoYnRNb0dTZHNLblVDUENsRGVuNGxuU3dwKzV5dERHbmgyUU9RRTZNUm03TTVuaVxuSktIYS9QQWpMdW1EMU9yL0VxZnRQWjg9XG4tLS0tLUVORCBQUklWQVRFIEtFWS0tLS0tXG4iLAogICJjbGllbnRfZW1haWwiOiAicHJvZC1rOHNzYW5kcmEtc2FAZ2NwLXRlY2hwdWJzLmlhbS5nc2VydmljZWFjY291bnQuY29tIiwKICAiY2xpZW50X2lkIjogIjExNDcwMDYyNDg5NzgwMDIzMTkwNiIsCiAgImF1dGhfdXJpIjogImh0dHBzOi8vYWNjb3VudHMuZ29vZ2xlLmNvbS9vL29hdXRoMi9hdXRoIiwKICAidG9rZW5fdXJpIjogImh0dHBzOi8vb2F1dGgyLmdvb2dsZWFwaXMuY29tL3Rva2VuIiwKICAiYXV0aF9wcm92aWRlcl94NTA5X2NlcnRfdXJsIjogImh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL29hdXRoMi92MS9jZXJ0cyIsCiAgImNsaWVudF94NTA5X2NlcnRfdXJsIjogImh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL3JvYm90L3YxL21ldGFkYXRhL3g1MDkvcHJvZC1rOHNzYW5kcmEtc2ElNDBnY3AtdGVjaHB1YnMuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20iCn0K kind: Secret metadata: creationTimestamp: "2021-07-08T19:13:29Z" managedFields:

And checking medusa container in one of the C* pods:

k logs prod-k8ssandra-dc1-us-central1-f-sts-0 -c medusa MEDUSA_MODE = GRPC sleeping for 0 sec Starting Medusa gRPC service /home/cassandra/.local/lib/python3.6/site-packages/requests/init.py:91: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) INFO:root:Init service [2021-07-08 19:15:15,866] INFO: Init service DEBUG:root:Loading storage_provider: google_storage [2021-07-08 19:15:15,866] DEBUG: Loading storage_provider: google_storage INFO:libcloud.common.google:Failed to read cached auth token from file "/home/cassandra/.google_libcloud_auth.prod-k8ssandra-sa@gcp-techpubs.iam.gserviceaccount.com": [Errno 2] No such file or directory: '/home/cassandra/.google_libcloud_auth.prod-k8ssandra-sa@gcp-techpubs.iam.gserviceaccount.com' [2021-07-08 19:15:15,867] INFO: Failed to read cached auth token from file "/home/cassandra/.google_libcloud_auth.prod-k8ssandra-sa@gcp-techpubs.iam.gserviceaccount.com": [Errno 2] No such file or directory: '/home/cassandra/.google_libcloud_auth.prod-k8ssandra-sa@gcp-techpubs.iam.gserviceaccount.com' DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): accounts.google.com:443 [2021-07-08 19:15:15,903] DEBUG: Starting new HTTPS connection (1): accounts.google.com:443 DEBUG:urllib3.connectionpool:https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None [2021-07-08 19:15:15,938] DEBUG: https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): storage.googleapis.com:443 [2021-07-08 19:15:15,942] DEBUG: Starting new HTTPS connection (1): storage.googleapis.com:443 DEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 "HEAD /prod-k8ssandra-storage-bucket HTTP/1.1" 200 0 [2021-07-08 19:15:16,017] DEBUG: https://storage.googleapis.com:443 "HEAD /prod-k8ssandra-storage-bucket HTTP/1.1" 200 0 INFO:root:Starting server. Listening on port 50051. [2021-07-08 19:15:17,289] INFO: Starting server. Listening on port 50051.

johnsmartco commented 3 years ago

Updating my related PR, https://github.com/k8ssandra/k8ssandra/pull/940, with modified commands (gke install topic first, then others that show the terraform output, added perf benchmark guidance, etc).

johnsmartco commented 3 years ago

Doc updated with -raw workaround and related edits.