search

lucky-sideburn / kubeinvaders

Gamified Chaos Engineering Tool for Kubernetes
Apache License 2.0
1.02k stars 127 forks source link

Aliens stop appearing after setting INSECURE_ENDPOINT=true #80

Closed InfoAge1985 closed 5 months ago

InfoAge1985 commented 5 months ago

Hello again, I'm running KubeInvaders via MiniKube (per the distribution instructions etc.) under a fairly huge Ubuntu Linux distribution under AWS (not EKS). kubecurl.log kube.log The aliens show ok and are promptly killed off. Two metrics variables are missing (chaos_jobs_node_count{node=workernode01}, and chaos_node_jobs_total). So, I use the following command to (what I believe I should do) to get them to appear: $ kubectl set env deployment/kubeinvaders INSECURE_ENDPOINT=true -n kubeinvaders Then, the app runs, but the aliens no longer appear. I have attached 2 log files per your instructions: kube.log and kubecurl.log Any assistance is appreciated, and Thank you!

lucky-sideburn commented 5 months ago

Hi @InfoAge1985

Can you please send to me all environment variables of the deployment of kubeinvaders?

Please open developer tools of the browser for watch console error and network requests during the game mode.

Thanks Eugenio

InfoAge1985 commented 5 months ago

I'm working on getting my Chrome browser (running via VNC) into developer mode, but here is the env:

$ kubectl -n kubeinvaders exec kubeinvaders-649d865df9-xkt7g -- env

PATH=/usr/local/openresty/nginx/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=kubeinvaders-649d865df9-xkt7g INSECURE_ENDPOINT=true ENDPOINT=kubeinvaders.local NAMESPACE=ns-1 ALIENPROXIMITY=10 HITSLIMIT=1 UPDATETIME=0.5 KUBEINVADERS_SERVICE_PORT_HTTP=80 KUBEINVADERS_PORT_80_TCP_PORT=80 KUBERNETES_SERVICE_PORT=443 KUBERNETES_PORT_443_TCP_PORT=443 KUBEINVADERS_PORT=tcp://10.96.138.78:80 KUBEINVADERS_PORT_80_TCP_PROTO=tcp KUBERNETES_SERVICE_HOST=10.96.0.1 KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT=tcp://10.96.0.1:443 KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 KUBEINVADERS_PORT_80_TCP_ADDR=10.96.138.78 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBEINVADERS_SERVICE_HOST=10.96.138.78 KUBEINVADERS_SERVICE_PORT=80 KUBEINVADERS_PORT_80_TCP=tcp://10.96.138.78:80 KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443 NGINX_VERSION=1.24.0 NJS_VERSION=0.7.12 PKG_RELEASE=1~bullseye HOME=/root

lucky-sideburn commented 5 months ago

Please do

curl -vvv https://kubeinvaders.local/kube/pods?action=list&namespace=ns-1

and

curl -vvv http://kubeinvaders.local/kube/pods?action=list&namespace=ns-1

Thanks!

InfoAge1985 commented 5 months ago

$ curl -vvv https://kubeinvaders.local/kube/pods?action=list&namespace=ns-1

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

[1]+ Exit 60 curl -vvv https://kubeinvaders.local/kube/pods?action=list

$ curl -vvv http://kubeinvaders.local/kube/pods?action=list&namespace=ns-1

[1]+ Done curl -vvv http://kubeinvaders.local/kube/pods?action=list

On Jun 20, 2024, at 12:27 PM, Eugenio Marzo @.***> wrote:

curl -vvv http://kubeinvaders.local/kube/pods?action=list&namespace=ns-1

lucky-sideburn commented 5 months ago

Uhm.. the redirect 308 is strange.. Below the output of my curl on secure endpoint. For you it should work with http instead of https. Can you add -L to the curl with https and -k to the curl with https?

[eugenio@DESKTOP-BUVN7L3.localdomain]:~/WORK$ >> curl "https://kubeinvaders.platformengineering.it/kube/pods?action=list&namespace=namespace1"
{"items":[{"status":"ready","name":"foobar-deployment-5995b6f5db-5d8qw"},{"status":"ready","name":"foobar-deployment-5995b6f5db-6c4w8"},{"status":"ready","name":"foobar-deployment-5995b6f5db-6x9r4"},{"status":"ready","name":"foobar-deployment-5995b6f5db-mrx97"},{"status":"ready","name":"foobar-deployment-5995b6f5db-tgjs8"},{"status":"ready","name":"nginx-deployment-5995b6f5db-4b6rn"},{"status":"ready","name":"nginx-deployment-5995b6f5db-6tsn6"},{"status":"ready","name":"nginx-deployment-5995b6f5db-kh2hc"},{"status":"ready","name":"nginx-deployment-5995b6f5db-vjdwg"},{"status":"ready","name":"nginx-deployment-5995b6f5db-zhbhx"}]}
[eugenio@DESKTOP-BUVN7L3.localdomain]:~/WORK$ >> curl  "http://kubeinvaders.platformengineering.it/kube/pods?action=list&namespace=namespace1"
[eugenio@DESKTOP-BUVN7L3.localdomain]:~/WORK$ >>
InfoAge1985 commented 5 months ago

$ curl -vvv -L https://kubeinvaders.local/kube/pods?action=list&namespace=ns-1

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

[1]+ Exit 60 curl -vvv -L https://kubeinvaders.local/kube/pods?action=list

=================================

$ curl -vvv -k https://kubeinvaders.local/kube/pods?action=list&namespace=ns-1

[1]+ Done curl -vvv -k https://kubeinvaders.local/kube/pods?action=list

On Jun 20, 2024, at 12:46 PM, Eugenio Marzo @.***> wrote:

Uhm.. the redirect 308 is strange.. Below the output of my curl on secure endpoint. For you it should work with http instead of https. Can you add -L to the curl with https and -k to the curl with https?

@.:~/WORK$ >> curl "https://kubeinvaders.platformengineering.it/kube/pods?action=list&namespace=namespace1" {"items":[{"status":"ready","name":"foobar-deployment-5995b6f5db-5d8qw"},{"status":"ready","name":"foobar-deployment-5995b6f5db-6c4w8"},{"status":"ready","name":"foobar-deployment-5995b6f5db-6x9r4"},{"status":"ready","name":"foobar-deployment-5995b6f5db-mrx97"},{"status":"ready","name":"foobar-deployment-5995b6f5db-tgjs8"},{"status":"ready","name":"nginx-deployment-5995b6f5db-4b6rn"},{"status":"ready","name":"nginx-deployment-5995b6f5db-6tsn6"},{"status":"ready","name":"nginx-deployment-5995b6f5db-kh2hc"},{"status":"ready","name":"nginx-deployment-5995b6f5db-vjdwg"},{"status":"ready","name":"nginx-deployment-5995b6f5db-zhbhx"}]} @.:~/WORK$ >> curl "http://kubeinvaders.platformengineering.it/kube/pods?action=list&namespace=namespace1" @.***:~/WORK$ >> — Reply to this email directly, view it on GitHub https://github.com/lucky-sideburn/kubeinvaders/issues/80#issuecomment-2181126580, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB6K3YJPUXCGSQ2ZYX5YJ2TZIMBPFAVCNFSM6AAAAABJUAD5E2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBRGEZDMNJYGA. You are receiving this because you were mentioned.

InfoAge1985 commented 5 months ago

And the following is for the http ones (as they had the 308’s on them):

$ curl -vvv -L http://kubeinvaders.local/kube/pods?action=list&namespace=ns-1

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

[1]+ Exit 60 curl -vvv -L http://kubeinvaders.local/kube/pods?action=list

=================================

$ curl -vvv -k http://kubeinvaders.local/kube/pods?action=list&namespace=ns-1

[1]+ Done curl -vvv -k http://kubeinvaders.local/kube/pods?action=list

On Jun 20, 2024, at 1:26 PM, D. Tim Clark @.***> wrote:

$ curl -vvv -L https://kubeinvaders.local/kube/pods?action=list&namespace=ns-1

  • Trying 10.211.55.70:443...
  • Connected to kubeinvaders.local (10.211.55.70) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • CAfile: /etc/ssl/certs/ca-certificates.crt
  • CApath: /etc/ssl/certs
  • TLSv1.0 (OUT), TLS header, Certificate Status (22):
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS header, Certificate Status (22):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS header, Finished (20):
  • TLSv1.2 (IN), TLS header, Supplemental data (23):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.2 (IN), TLS header, Supplemental data (23):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (OUT), TLS header, Unknown (21):
  • TLSv1.3 (OUT), TLS alert, unknown CA (560):
  • SSL certificate problem: self-signed certificate
  • Closing connection 0 curl: (60) SSL certificate problem: self-signed certificate More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

[1]+ Exit 60 curl -vvv -L https://kubeinvaders.local/kube/pods?action=list

=================================

$ curl -vvv -k https://kubeinvaders.local/kube/pods?action=list&namespace=ns-1

  • Trying 10.211.55.70:443...
  • Connected to kubeinvaders.local (10.211.55.70) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • TLSv1.0 (OUT), TLS header, Certificate Status (22):
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS header, Certificate Status (22):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS header, Finished (20):
  • TLSv1.2 (IN), TLS header, Supplemental data (23):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.2 (IN), TLS header, Supplemental data (23):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS header, Supplemental data (23):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.2 (IN), TLS header, Supplemental data (23):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.2 (OUT), TLS header, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.2 (OUT), TLS header, Supplemental data (23):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
  • start date: Jun 20 13:26:27 2024 GMT
  • expire date: Jun 20 13:26:27 2025 GMT
  • issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
  • SSL certificate verify result: self-signed certificate (18), continuing anyway.
  • Using HTTP2, server supports multiplexing
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • TLSv1.2 (OUT), TLS header, Supplemental data (23):
  • TLSv1.2 (OUT), TLS header, Supplemental data (23):
  • TLSv1.2 (OUT), TLS header, Supplemental data (23):
  • Using Stream ID: 1 (easy handle 0x5fa7925e0580)
  • TLSv1.2 (OUT), TLS header, Supplemental data (23):

    GET /kube/pods?action=list HTTP/2 Host: kubeinvaders.local user-agent: curl/7.81.0 accept: /

  • TLSv1.2 (IN), TLS header, Supplemental data (23):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • TLSv1.2 (IN), TLS header, Supplemental data (23):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • old SSL session ID is stale, removing
  • TLSv1.2 (IN), TLS header, Supplemental data (23):
  • Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
  • TLSv1.2 (OUT), TLS header, Supplemental data (23):
  • TLSv1.2 (IN), TLS header, Supplemental data (23): < HTTP/2 500 < date: Thu, 20 Jun 2024 16:52:03 GMT < content-type: text/html < content-length: 183 < access-control-allow-origin: * < access-control-allow-methods: GET, POST, OPTIONS < access-control-allow-headers: DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range < access-control-expose-headers: Content-Length,Content-Range < 500 Internal Server Error

    500 Internal Server Error

    openresty/1.25.3.1
  • TLSv1.2 (IN), TLS header, Supplemental data (23):
  • Connection #0 to host kubeinvaders.local left intact

[1]+ Done curl -vvv -k https://kubeinvaders.local/kube/pods?action=list

On Jun 20, 2024, at 12:46 PM, Eugenio Marzo @.***> wrote:

Uhm.. the redirect 308 is strange.. Below the output of my curl on secure endpoint. For you it should work with http instead of https. Can you add -L to the curl with https and -k to the curl with https?

@.:~/WORK$ >> curl "https://kubeinvaders.platformengineering.it/kube/pods?action=list&namespace=namespace1" {"items":[{"status":"ready","name":"foobar-deployment-5995b6f5db-5d8qw"},{"status":"ready","name":"foobar-deployment-5995b6f5db-6c4w8"},{"status":"ready","name":"foobar-deployment-5995b6f5db-6x9r4"},{"status":"ready","name":"foobar-deployment-5995b6f5db-mrx97"},{"status":"ready","name":"foobar-deployment-5995b6f5db-tgjs8"},{"status":"ready","name":"nginx-deployment-5995b6f5db-4b6rn"},{"status":"ready","name":"nginx-deployment-5995b6f5db-6tsn6"},{"status":"ready","name":"nginx-deployment-5995b6f5db-kh2hc"},{"status":"ready","name":"nginx-deployment-5995b6f5db-vjdwg"},{"status":"ready","name":"nginx-deployment-5995b6f5db-zhbhx"}]} @.:~/WORK$ >> curl "http://kubeinvaders.platformengineering.it/kube/pods?action=list&namespace=namespace1" @.***:~/WORK$ >> — Reply to this email directly, view it on GitHub https://github.com/lucky-sideburn/kubeinvaders/issues/80#issuecomment-2181126580, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB6K3YJPUXCGSQ2ZYX5YJ2TZIMBPFAVCNFSM6AAAAABJUAD5E2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBRGEZDMNJYGA. You are receiving this because you were mentioned.

lucky-sideburn commented 5 months ago

Hi @InfoAge1985 ok maybe we found the problem.. INSECURE_ENDPOINT is only of development purpose and is not supported if you use the helm chart. The reason is that in https://github.com/lucky-sideburn/kubeinvaders/blob/master/helm-charts/kubeinvaders/templates/ingress.yaml the default ingress use TLS and this is why you have the redirect 308.

Can you try to delete the tls part from the ingress?

InfoAge1985 commented 5 months ago

I have deleted the tls from the ingress.yaml file, and then removed the tis {} from the values.yaml in its parent directory . . . and also removed the INSECURE_ENDPOINT=true . . . and iterated through each of these combinations.

I am still getting the following:

[1]+ Done curl -vvv http://kubeinvaders.local/kube/pods?action=list

=====================

If you have a working ingress.yaml file, please forward it in a reply . . .

Thank you for all your help . . .

On Jun 20, 2024, at 4:16 PM, Eugenio Marzo @.***> wrote:

Hi @InfoAge1985 https://github.com/InfoAge1985 ok maybe we found the problem.. INSECURE_ENDPOINT is only of development purpose and is not supported if you use the helm chart. The reason is that in https://github.com/lucky-sideburn/kubeinvaders/blob/master/helm-charts/kubeinvaders/templates/ingress.yaml the default ingress use TLS and this is why you have the redirect 308.

Can you try to delete the tls part from the ingress?

— Reply to this email directly, view it on GitHub https://github.com/lucky-sideburn/kubeinvaders/issues/80#issuecomment-2181487215, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB6K3YN2QP67E2QDH5FSJ73ZIM2ALAVCNFSM6AAAAABJUAD5E2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBRGQ4DOMRRGU. You are receiving this because you were mentioned.

lucky-sideburn commented 5 months ago

Hi @InfoAge1985 Sorry for too many questions :D Did you also then run helm upgrade or install so the actually active ingress has TLS disabled?

InfoAge1985 commented 5 months ago

First, thank you for all you are doing to help me make this work. I tried manually inserting a kubeinvaders-1.9.7.tgz file into the helm repository . . . once with the "tls" section removed from the ingress.yaml file, then another with the "tls {}" line removed from the values.yaml file ... neither approach worked. I am not proficient with helm enough to get this to work, so I'd appreciate clear instruction on how to do this. Again, thank you!

InfoAge1985 commented 5 months ago

Ok . . . I'm going to stop working on this as I'm asking too much of you. I appreciate all you have done to try to help me and I wish you only the best.