search

pixie-io / pixie

Instant Kubernetes-Native Application Observability
https://px.dev
Apache License 2.0
5.47k stars 422 forks source link

Setup Pixie locally with Colima fails to initialise ElasticSearch #1858

Open dcfranca opened 5 months ago

dcfranca commented 5 months ago

Hello team, First, I'd like to say that I'm very excited to try out Pixie

I was trying to install locally on a local Kubernetes cluster on top of Colima, is it supported? I was not sure but I tried anyway as this is how the local development happens at my company

I'm running on a MacOS M3 Max with Sonoma 14.4

Colima version

colima version 0.6.8
git commit: 9b0809d0ed9ad3ff1e57c405f27324e6298ca04f

runtime: docker
arch: aarch64
client: v25.0.0
server: v24.0.7

Then I have the first issue on the Install Pixie Cloud section

A few pods in CrashLoopBackoff

postgres-56f89f7d58-vlmns                  1/1     Running            1 (49m ago)      75m
artifact-tracker-server-76ddc4d6bc-5d4dp   1/1     Running            1 (49m ago)      75m
config-manager-server-6d4767f49b-w8ddd     1/1     Running            1 (49m ago)      75m
hydra-7c757587bf-pxrh8                     2/2     Running            2 (49m ago)      75m
profile-server-7bdbb6b487-dmf96            1/1     Running            2 (49m ago)      75m
cron-script-server-6545944d64-tmgjp        1/1     Running            2 (49m ago)      75m
plugin-server-6fc5474d7b-kr2rw             1/1     Running            2 (49m ago)      75m
auth-server-54cd5cd974-rptf4               1/1     Running            2 (49m ago)      75m
kratos-5fd6cd6c7d-ltbmr                    2/2     Running            2 (49m ago)      75m
metrics-server-6dc964b448-prsfm            1/1     Running            2 (49m ago)      75m
project-manager-server-5d798ff9f6-sm2m5    1/1     Running            2 (49m ago)      75m
pl-nats-2                                  1/1     Running            1 (49m ago)      75m
pl-nats-1                                  1/1     Running            1 (49m ago)      75m
pl-nats-0                                  1/1     Running            1 (49m ago)      75m
vzmgr-server-6db898b648-dj8zl              1/1     Running            3 (48m ago)      75m
vzconn-server-6499968767-4zln8             1/1     Running            3 (48m ago)      75m
scriptmgr-server-5b468f58ff-xlrfk          1/1     Running            3 (48m ago)      75m
pl-elastic-es-data-0                       0/1     CrashLoopBackOff   23 (2m35s ago)   75m
pl-elastic-es-master-0                     0/1     CrashLoopBackOff   11 (2m9s ago)    33m
indexer-server-58f7846bb7-5hzj9            0/1     CrashLoopBackOff   25 (97s ago)     75m
api-server-5ccbc5b66c-gs78n                0/1     CrashLoopBackOff   26 (50s ago)     75m
cloud-proxy-7897b497cb-d6mdk               1/2     CrashLoopBackOff   32 (33s ago)     68m
pl-elastic-es-master-1                     0/1     Running            12 (5m6s ago)    36m
plugin-db-updater-job-bd6mf                0/1     PodInitializing    0                2s

It seems to be a cascade issue due to the elastic search pods crashing Logs from the elastic search:

sysctl vm.max_map_count = 262144
Stream closed EOF for plc/pl-elastic-es-master-0 (sysctl)
elasticsearch Exception in thread "main" java.io.IOException: Cannot run program "/usr/share/elasticsearch/jdk/bin/java": error=0, Failed to exec spawn helper.
elastic-internal-init-filesystem Starting init script
elastic-internal-init-filesystem Linking /mnt/elastic-internal/xpack-file-realm/users to /usr/share/elasticsearch/config/users
elastic-internal-init-filesystem Linking /mnt/elastic-internal/xpack-file-realm/roles.yml to /usr/share/elasticsearch/config/roles.yml
elastic-internal-init-filesystem Linking /mnt/elastic-internal/xpack-file-realm/users_roles to /usr/share/elasticsearch/config/users_roles
elastic-internal-init-filesystem Linking /mnt/elastic-internal/elasticsearch-config/elasticsearch.yml to /usr/share/elasticsearch/config/elasticsearch.yml
elasticsearch     at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
elasticsearch     at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
elasticsearch     at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:111)
elasticsearch     at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88)
elasticsearch     at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
elasticsearch     at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95)
elasticsearch Caused by: java.io.IOException: error=0, Failed to exec spawn helper.
elasticsearch     at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
elasticsearch     at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:319)
elasticsearch     at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:250)
elasticsearch     at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
elasticsearch     ... 5 more
elastic-internal-init-filesystem Linking /mnt/elastic-internal/unicast-hosts/unicast_hosts.txt to /usr/share/elasticsearch/config/unicast_hosts.txt
elastic-internal-init-filesystem File linking duration: 1 sec.
elastic-internal-init-filesystem Copying /usr/share/elasticsearch/config/* to /mnt/elastic-internal/elasticsearch-config-local/
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/elasticsearch.yml' -> '/mnt/elastic-internal/elasticsearch-config-local/elasticsearch.yml'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/ca.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/tls.key' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/tls.key'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/tls.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/tls.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..data' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..data'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..2024_03_15_18_24_41.3325295662' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2024_03_15_18_24_41.3325295662'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..2024_03_15_18_24_41.3325295662/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2024_03_15_18_24_41.3325295662/ca.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..2024_03_15_18_24_41.3325295662/tls.key' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2024_03_15_18_24_41.3325295662/tls.key'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..2024_03_15_18_24_41.3325295662/tls.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2024_03_15_18_24_41.3325295662/tls.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/jvm.options' -> '/mnt/elastic-internal/elasticsearch-config-local/jvm.options'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/log4j2.properties' -> '/mnt/elastic-internal/elasticsearch-config-local/log4j2.properties'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/role_mapping.yml' -> '/mnt/elastic-internal/elasticsearch-config-local/role_mapping.yml'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/roles.yml' -> '/mnt/elastic-internal/elasticsearch-config-local/roles.yml'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/ca.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs/..data' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/..data'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs/..2024_03_15_18_24_41.3095262462' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/..2024_03_15_18_24_41.3095262462'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs/..2024_03_15_18_24_41.3095262462/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/..2024_03_15_18_24_41.3095262462/ca.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/unicast_hosts.txt' -> '/mnt/elastic-internal/elasticsearch-config-local/unicast_hosts.txt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/users' -> '/mnt/elastic-internal/elasticsearch-config-local/users'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/users_roles' -> '/mnt/elastic-internal/elasticsearch-config-local/users_roles'
elastic-internal-init-filesystem Empty dir /usr/share/elasticsearch/plugins
elastic-internal-init-filesystem Copying /usr/share/elasticsearch/bin/* to /mnt/elastic-internal/elasticsearch-bin-local/
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-certgen' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-certgen'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-certutil' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-certutil'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-cli' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-cli'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-croneval' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-croneval'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-env'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-env-from-file' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-env-from-file'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-keystore' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-keystore'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-migrate' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-migrate'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-node' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-node'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-plugin' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-plugin'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-saml-metadata' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-saml-metadata'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-setup-passwords' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-setup-passwords'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-shard' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-shard'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-sql-cli' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-sql-cli'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-sql-cli-7.6.0.jar' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-sql-cli-7.6.0.jar'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-syskeygen' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-syskeygen'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-users' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-users'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/x-pack-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/x-pack-env'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/x-pack-security-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/x-pack-security-env'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/x-pack-watcher-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/x-pack-watcher-env'
elastic-internal-init-filesystem Files copy duration: 1 sec.
elastic-internal-init-filesystem chowning /usr/share/elasticsearch/data to elasticsearch:elasticsearch
elastic-internal-init-filesystem ownership of '/usr/share/elasticsearch/data' retained as elasticsearch:elasticsearch
elastic-internal-init-filesystem chowning /usr/share/elasticsearch/logs to elasticsearch:elasticsearch
elastic-internal-init-filesystem changed ownership of '/usr/share/elasticsearch/logs' from root:root to elasticsearch:elasticsearch
elastic-internal-init-filesystem chown duration: 1 sec.
elastic-internal-init-filesystem waiting for the transport certificates (/mnt/elastic-internal/transport-certificates/pl-elastic-es-master-0.tls.key)
elastic-internal-init-filesystem wait duration: 0 sec.
elastic-internal-init-filesystem Linking /usr/share/elasticsearch/config/transport-certs/pl-elastic-es-master-0.tls.crt to /mnt/elastic-internal/elasticsearch-config-local/node-transport-cert/transport.tls.crt
elastic-internal-init-filesystem Linking /usr/share/elasticsearch/config/transport-certs/pl-elastic-es-master-0.tls.crt to /mnt/elastic-internal/elasticsearch-config-local/node-transport-cert/transport.tls.crt
elastic-internal-init-filesystem Certs linking duration: 1 sec.
elastic-internal-init-filesystem Init script successful
elastic-internal-init-filesystem Script duration: 4 sec.
Stream closed EOF for plc/pl-elastic-es-master-0 (elastic-internal-init-filesystem)
Stream closed EOF for plc/pl-elastic-es-master-0 (elasticsearch)

I tried to continue the setup, but then I get other errors, I believe this one should be fixed first For example, running dev_dns_updater gets stuck after typing the password

Any idea what could be causing ElasticSearch failure? What I found googling was related to not working on Apple Silicion, but it seems that since then it was fixed: https://stackoverflow.com/questions/65962810/m1-mac-issue-bringing-up-elasticsearch-cannot-run-jdk-bin-java

dcfranca commented 5 months ago

Any idea?

dcfranca commented 5 months ago

Anyone?

JamesMBartlett commented 5 months ago

The version of Elasticsearch in the stackoverflow you linked is 7.10.2. Pixie currently uses 7.6. It's possible this is an issue with elasticsearch that got fixed between 7.6 and 7.10.2.

You could try changing this line https://github.com/pixie-io/pixie/blob/cf88e332875094fdf4cc423f582c23c1fe957729/k8s/cloud_deps/base/elastic/cluster/elastic_cluster.yaml#L8 to use the image suggested in the stackoverflow post.

dcfranca commented 4 months ago

@JamesMBartlett If I change the image to the one from the StackOverflow answer I get 

chroot: failed to set supplemental groups: Operation not permitted

Not sure if it might be because the one from the SO answer is a public image, while the Pixie one is a patched image

dcfranca commented 4 months ago

Ok, I managed to make it work removing the securityContext capabilities restrictions, which is not ideal, but it should work for local tests, if you know a Pixie patched image that I can use instead it would help

             allowPrivilegeEscalation: false                                                                                                                │   • (×) Install SWE app
-            capabilities:                                                                                                                                  │   • (×) Check Active OrderUp
-              add:                                                                                                                                         │   • (×) List deployments in execution (spin)
-              - SYS_CHROOT                                                                                                                                 │   • (×) Grab latest build in Teran
-              - SETUID                                                                                                                                     │
-              drop:                                                                                                                                        │   • (×) OrderUp: Python Client
-              - ALL