Closed jdmulloy closed 1 year ago
chadlwilson
commented
1 year ago That sounds odd. I'm sure ive used it off the shelf recently with default configs and not had issues but can check.
Can you report your specific chart, GoCD and plugin versions to avoid confusion?
jdmulloy
commented
1 year ago I can check. I'll admit the setup is a little weird because it's our internal minikube dev setup. We're also using our own GoCD image based on the official one. This was hard to debug because it's hard to see the logs from the preconfigure script.
GoCD version 21.4.0 Elastic Agent Version 3.4.0-196
Our dockerfile for GoCD server
USER root
RUN cd /home/go \
&& curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
&& unzip awscliv2.zip \
&& ./aws/install
ENTRYPOINT ["/docker-entrypoint.sh"]
USER go```Chart version: 1.39.3
jdmulloy
commented
1 year ago I just tried it with a script locally and what I thought was failing worked. Not sure why. I guess maybe it's not broken. It's hard to debug because the script output doesn't go to the kubernetes logs and if it fails the container stops and I can't read the logs easily because I can't kubectl exec on a dead pod.
jdmulloy
commented
1 year ago I take that back, I tried it with MaxMemory and it worked, if I remove max memory it fails.
Fails:
#!/bin/bash
set -x
SERVICE_ACCOUNT_PATH=/var/run/secrets/kubernetes.io/serviceaccount
KUBE_TOKEN=$(kubectl -n gocd exec --stdin --tty $(kubectl -n gocd get pods | grep gocd-server | awk '{print $1}') -- cat ${SERVICE_ACCOUNT_PATH}/token)
KUBERNETES_SERVICE_HOST=$(kubectl -n gocd exec --stdin --tty $(kubectl -n gocd get pods | grep gocd-server | awk '{print $1}') -- sh -c 'echo $KUBERNETES_SERVICE_HOST')
KUBERNETES_SERVICE_PORT_HTTPS=$(kubectl -n gocd exec --stdin --tty $(kubectl -n gocd get pods | grep gocd-server | awk '{print $1}') -- sh -c 'echo $KUBERNETES_SERVICE_PORT_HTTPS')
curl --fail -i -v -k 'https://gocd.development.example.com/go/api/elastic/profiles' \
-H 'Accept: application/vnd.go.cd+json' \
-H 'Content-Type: application/json' \
-X POST -d '{
"id": "demo-app",
"cluster_profile_id": "k8-cluster-profile",
"properties": [
{
"key": "Image",
"value": "gocd/gocd-agent-docker-dind:v21.3.0"
},
{
"key": "PodConfiguration",
"value": "apiVersion: v1\nkind: Pod\nmetadata:\n name: gocd-agent-{{ POD_POSTFIX }}\n labels:\n app: web\nspec:\n serviceAccountName: default\n containers:\n - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}\n image: gocd/gocd-agent-docker-dind:v21.3.0\n securityContext:\n privileged: true"
},
{
"key": "PodSpecType",
"value": "yaml"
},
{
"key": "Privileged",
"value": "true"
}
]
}'Works
#!/bin/bash
set -x
SERVICE_ACCOUNT_PATH=/var/run/secrets/kubernetes.io/serviceaccount
KUBE_TOKEN=$(kubectl -n gocd exec --stdin --tty $(kubectl -n gocd get pods | grep gocd-server | awk '{print $1}') -- cat ${SERVICE_ACCOUNT_PATH}/token)
KUBERNETES_SERVICE_HOST=$(kubectl -n gocd exec --stdin --tty $(kubectl -n gocd get pods | grep gocd-server | awk '{print $1}') -- sh -c 'echo $KUBERNETES_SERVICE_HOST')
KUBERNETES_SERVICE_PORT_HTTPS=$(kubectl -n gocd exec --stdin --tty $(kubectl -n gocd get pods | grep gocd-server | awk '{print $1}') -- sh -c 'echo $KUBERNETES_SERVICE_PORT_HTTPS')
curl --fail -i -v -k 'https://gocd.development.example.com/go/api/elastic/profiles' \
-H 'Accept: application/vnd.go.cd+json' \
-H 'Content-Type: application/json' \
-X POST -d '{
"id": "demo-app",
"cluster_profile_id": "k8-cluster-profile",
"properties": [
{
"key": "Image",
"value": "gocd/gocd-agent-docker-dind:v21.3.0"
},
{
"key": "PodConfiguration",
"value": "apiVersion: v1\nkind: Pod\nmetadata:\n name: gocd-agent-{{ POD_POSTFIX }}\n labels:\n app: web\nspec:\n serviceAccountName: default\n containers:\n - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}\n image: gocd/gocd-agent-docker-dind:v21.3.0\n securityContext:\n privileged: true"
},
{
"key": "PodSpecType",
"value": "yaml"
},
{
"key": "MaxMemory",
"value": "1G"
},
{
"key": "Privileged",
"value": "true"
}
]
}'
+ SERVICE_ACCOUNT_PATH=/var/run/secrets/kubernetes.io/serviceaccount
+++ kubectl -n gocd get pods
+++ grep gocd-server
+++ awk '{print $1}'
++ kubectl -n gocd exec --stdin --tty development-gocd-server-665879b8b7-dbcmk -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
Defaulted container "gocd-server" out of: gocd-server, vault-authenticator (init), ssh-key-render (init), opsgenie-secret-render (init), users-render (init), gocd-analytics-plugin-js-init-container (init)
+ KUBE_TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6ImQ5R3NVa3llcnBKbmZpUTZRbHBYQVhuS1NfbFZSZDkxTnVNbnRaNkp6emMifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjk4ODU2NzMyLCJpYXQiOjE2NjczMjA3MzIsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJnb2NkIiwicG9kIjp7Im5hbWUiOiJkZXZlbG9wbWVudC1nb2NkLXNlcnZlci02NjU4NzliOGI3LWRiY21rIiwidWlkIjoiZDcyM2RiNmUtZTYzZi00YTA4LTk2OWEtZDc1YmM0NDczNDdmIn0sInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJkZXZlbG9wbWVudC1nb2NkIiwidWlkIjoiMGI3ZTJmZGItOWQxYy00NzdlLWI4YWUtNzIzNDFmMjJiZGZkIn0sIndhcm5hZnRlciI6MTY2NzMyNDMzOX0sIm5iZiI6MTY2NzMyMDczMiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmdvY2Q6ZGV2ZWxvcG1lbnQtZ29jZCJ9.SIwPPOdt4-4K6-fuwAz_ZTbcdevNFt7GghylwEdW346a0mn2Eu8x6FX9bstGV23snB6lAt2Z2eXEJR9CuPLbhO4Suyg3rjt8fou7GYZCgf3xRKqb42tzcfRIplc-OZVM0TjTU3am-Qy5IU4m2b3GQkkmd_bUryGOCmx470zGqGQK3m2sBsFX95bD5g2Y7pVobfcivu_IPX4AEAsQ_3_AZ3j4jZ_gw74HEHRRx0-h32xHeAknzapHpgPbFVqNHXH2fBM18geLEZ76_8i4dVJyz5TpclU_EA8dTpfb3lvJp_IklTZulwQ9yvpPol5HEEamKIFFW3eYV0t3MfaVqWXuig
+++ kubectl -n gocd get pods
+++ grep gocd-server
+++ awk '{print $1}'
++ kubectl -n gocd exec --stdin --tty development-gocd-server-665879b8b7-dbcmk -- sh -c 'echo $KUBERNETES_SERVICE_HOST'
Defaulted container "gocd-server" out of: gocd-server, vault-authenticator (init), ssh-key-render (init), opsgenie-secret-render (init), users-render (init), gocd-analytics-plugin-js-init-container (init)
+ KUBERNETES_SERVICE_HOST=$'10.96.0.1\r'
+++ kubectl -n gocd get pods
+++ grep gocd-server
+++ awk '{print $1}'
++ kubectl -n gocd exec --stdin --tty development-gocd-server-665879b8b7-dbcmk -- sh -c 'echo $KUBERNETES_SERVICE_PORT_HTTPS'
Defaulted container "gocd-server" out of: gocd-server, vault-authenticator (init), ssh-key-render (init), opsgenie-secret-render (init), users-render (init), gocd-analytics-plugin-js-init-container (init)
+ KUBERNETES_SERVICE_PORT_HTTPS=$'443\r'
+ curl --fail -i -v -k https://gocd.development.example.com/go/api/elastic/profiles -H 'Accept: application/vnd.go.cd+json' -H 'Content-Type: application/json' -X POST -d '{
"id": "demo-app",
"cluster_profile_id": "k8-cluster-profile",
"properties": [
{
"key": "Image",
"value": "gocd/gocd-agent-docker-dind:v21.3.0"
},
{
"key": "PodConfiguration",
"value": "apiVersion: v1\nkind: Pod\nmetadata:\n name: gocd-agent-{{ POD_POSTFIX }}\n labels:\n app: web\nspec:\n serviceAccountName: default\n containers:\n - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}\n image: gocd/gocd-agent-docker-dind:v21.3.0\n securityContext:\n privileged: true"
},
{
"key": "PodSpecType",
"value": "yaml"
},
{
"key": "Privileged",
"value": "true"
}
]
}'
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 192.168.39.23:443...
* TCP_NODELAY set
* Connected to gocd.development.example.com (192.168.39.23) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* start date: Nov 1 16:35:13 2022 GMT
* expire date: Nov 1 16:35:13 2023 GMT
* issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55ff426c02f0)
> POST /go/api/elastic/profiles HTTP/2
> Host: gocd.development.example.com
> user-agent: curl/7.68.0
> accept: application/vnd.go.cd+json
> content-type: application/json
> content-length: 719
>
* We are completely uploaded and fine
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* The requested URL returned error: 422
* stopped the pause stream!
* Connection #0 to host gocd.development.example.com left intact
curl: (22) The requested URL returned error: 422```I cannot replicate this, and the chart works fine OOTB with the chart-suggested plugin versions.
While I don't think there has been any change to MaxMemory validation (it's optional, and only relevant for non-YAML configuration anyway) - your plugin version is 3 years old and perhaps there is/was an API compatibility issue. Rather than dig further into the server and plugin logs etc - it's probably a more effective use of time to just validate the versions and use a more recent elastic agent plugin version at the very least.
If your GoCD version is 21.4.0 there should be no issue with using the latest version of the plugin from a compatibility standpoint.
The preconfigure script gets an error when trying to apply the elastic kubernetes agent profile because it doesn't set a value for MaxMemory which is required. I'm guessing GoCD or the elastic agent plugin changed at some point to require that field.