search

instana / onprem-distkit

A collection of configurations that may be useful for operators of distributed Instana deployments
4 stars 2 forks source link

Release-207 intergration test: agent no data in instana UI #35

Closed yaoyao12138 closed 3 years ago

yaoyao12138 commented 3 years ago

On kind cluster, all instana pods are ruuning and start the agent, but there is no data from agent in instana UI.
image

root@yao-instana-vm-64-instana-main:/opt/instana/agent/etc/instana# systemctl status instana-agent
● instana-agent.service - "Instana(tm) agent."
     Loaded: loaded (/lib/systemd/system/instana-agent.service; disabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/instana-agent.service.d
             └─custom-environment.conf
     Active: active (running) since Wed 2021-10-27 01:33:43 PDT; 35min ago
   Main PID: 1684535 (java)
     Status: "Agent is available"
      Tasks: 84 (limit: 77081)
     Memory: 628.3M
     CGroup: /system.slice/instana-agent.service
             ├─1684535 /opt/instana/agent/jvm/bin/java -XX:+UnlockDiagnosticVMOptions -XX:+UseSerialGC -Xmn32m -Xss512k -XX:InitialCodeCacheSize=8m -XX:CodeCacheExpansionSize=64k -XX:ReservedCodeCacheSize=16>
             └─1690481 /opt/instana/agent/data/repo/com/instana/ebpf-sensor/0.1.6/ebpf-sensor-0.1.6.bin

Oct 27 01:33:19 yao-instana-vm-64-instana-main.fyre.ibm.com systemd[1]: Starting "Instana(tm) agent."...
Oct 27 01:33:19 yao-instana-vm-64-instana-main.fyre.ibm.com karaf[1684535]: The memory calculator has not been used, and neither the 'JAVA_MAX_MEM' nor 'JAVA_OPTS' environment variables specify a maximum hea>
Oct 27 01:33:43 yao-instana-vm-64-instana-main.fyre.ibm.com systemd[1]: Started "Instana(tm) agent.".

There is a lot of exceptions in acceptor pod. 

kubectl logs acceptor-65dfd66896-bsmvq -n instana-core
"dw-16" #16 prio=5 os_prio=0 cpu=2.16ms elapsed=34.27s tid=0x00007f251d22b800 nid=0x3d runnable  [0x00007f24f8bba000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPoll.wait(java.base@11.0.12/Native Method)
        at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12/EPollSelectorImpl.java:120)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12/SelectorImpl.java:124)
        - locked <0x00000000ffae32f0> (a sun.nio.ch.Util$2)
        - locked <0x00000000ffae31a0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(java.base@11.0.12/SelectorImpl.java:141)
        at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183)
        at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:190)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$729/0x0000000100711c40.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
        at java.lang.Thread.run(java.base@11.0.12/Thread.java:829)

"dw-17" #17 prio=5 os_prio=0 cpu=135.90ms elapsed=34.27s tid=0x00007f251d22c800 nid=0x3e waiting on condition  [0x00007f24f8b77000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.12/Native Method)
        - parking to wait for  <0x00000000ff4945f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.12/LockSupport.java:234)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.12/AbstractQueuedSynchronizer.java:2123)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:973)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1023)
        at java.lang.Thread.run(java.base@11.0.12/Thread.java:829)

"dw-18" #18 prio=5 os_prio=0 cpu=0.40ms elapsed=34.27s tid=0x00007f251d22e800 nid=0x3f runnable  [0x00007f24f8b34000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPoll.wait(java.base@11.0.12/Native Method)
        at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12/EPollSelectorImpl.java:120)
yaoyao12138 commented 3 years ago

After cleaned up all db and instana data, then re-start db service and instana, this question still exist.
All pods in the ns instana-core and instana-units have many java exceptions.
Try to use k3s instead of kind to launch instana.

yaoyao12138 commented 3 years ago

Use k3s to launch instana and db clusters, when logged in, the UI crashed. image

root@yao-instana-vm-64-instana-k8s-env:~# k3s kubectl get po
...
instana-core       groundskeeper-8646b55c49-5ttht                             1/1     Running   0          3m52s
instana-units      tu-instana-prod-ui-backend-85cccd9949-qfstt                1/1     Running   0          3m24s
instana-units      ui-client-5f6bd97747-8svbd                                 1/1     Running   0          3m5s
instana-units      tu-instana-prod-appdata-processor-b6dcc9997-vvqlp          0/1     Pending   0          89s
root@yao-instana-vm-64-instana-k8s-env:~# k3s kubectl describe po tu-instana-prod-appdata-processor-b6dcc9997-vvqlp -n instana-units
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  86s   default-scheduler  0/1 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity rules, 1 node(s) didn't match pod anti-affinity rules.
  Warning  FailedScheduling  84s   default-scheduler  0/1 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity rules, 1 node(s) didn't match pod anti-affinity rules.

In the ui-client pod, the log is the same as which in the kind env. 

kubectl logs ui-client-5f6bd97747-8svbd -n instana-units
...
{"name":"@instana/collector","__in":1,"hostname":"ui-client-5f6bd97747-8svbd","pid":1,"module":"announceCycle/agentHostLookup","level":40,"msg":"Agent cannot be contacted via 127.0.0.1:42699 and default gateway cannot be determined. Scheduling reattempt of agent host lookup in 60000 millis.","time":"2021-10-29T02:34:55.484Z","v":0}

And other pod don't have exceptions still. 

root@yao-instana-vm-64-instana-k8s-env:~# k3s kubectl get po -A
NAMESPACE          NAME                                                       READY   STATUS    RESTARTS   AGE
kube-system        local-path-provisioner-5ff76fc89d-xl4dm                    1/1     Running   0          17h
kube-system        metrics-server-86cbb8457f-w48sb                            1/1     Running   0          17h
kube-system        coredns-7448499f4d-6knqb                                   1/1     Running   0          17h
default            ambassador-agent-69bdc547bd-bhztf                          1/1     Running   0          17h
default            svclb-ambassador-5bpvt                                     2/2     Running   0          17h
default            nfs-subdir-external-provisioner-7569bc74dc-npwtm           1/1     Running   0          17h
default            ambassador-6677db9456-5jlvk                                1/1     Running   0          17h
default            ambassador-6677db9456-7vn7b                                1/1     Running   0          17h
default            ambassador-6677db9456-npk47                                1/1     Running   0          17h
instana-operator   instana-selfhosted-operator-5b578876b-tglpr                1/1     Running   0          17h
instana-core       butler-6d7456f54-l8hmt                                     1/1     Running   0          17h
instana-core       accountant-86498cb68-vwsjx                                 1/1     Running   0          17h
instana-core       cashier-ingest-76684948bd-947nn                            1/1     Running   0          17h
instana-core       cashier-rollup-7c4ccc85d7-qxl7j                            1/1     Running   0          17h
instana-core       acceptor-79d566d599-5psh7                                  1/1     Running   0          17h
instana-core       eum-acceptor-6fc55df694-mz45d                              1/1     Running   0          17h
instana-core       eum-processor-7c878ccdb7-6zz7r                             1/1     Running   0          17h
instana-core       eum-health-processor-6f8b59bff8-6l8v4                      1/1     Running   0          17h
instana-core       appdata-health-processor-67dd86ff-v9vg6                    1/1     Running   0          17h
instana-core       sli-evaluator-78c6bb86-lsbmw                               1/1     Running   0          17h
instana-core       js-stack-trace-translator-5dfbc485c9-b6j88                 1/1     Running   0          17h
instana-core       appdata-writer-7447f869c4-mchc6                            1/1     Running   0          17h
instana-core       appdata-reader-77b667d9f-nxswn                             1/1     Running   0          17h
instana-core       serverless-acceptor-6db875466b-bcj6b                       1/1     Running   0          17h
instana-core       ingress-core-7c767b65-zg8rt                                1/1     Running   0          17h
instana-units      ingress-65d9589c5-r2768                                    1/1     Running   0          17h
instana-units      tu-instana-prod-appdata-legacy-converter-87d764f9c-jzhzm   1/1     Running   0          17h
instana-units      tu-instana-prod-filler-58b4fcc655-lsrxc                    1/1     Running   0          17h
instana-units      tu-instana-prod-processor-cfd5ffc5d-mgtkf                  1/1     Running   0          17h
instana-units      tu-instana-prod-issue-tracker-597cfb9db7-qjdgd             1/1     Running   0          17h
instana-core       groundskeeper-8646b55c49-5ttht                             1/1     Running   0          17h
instana-units      tu-instana-prod-ui-backend-85cccd9949-qfstt                1/1     Running   0          17h
instana-units      ui-client-5f6bd97747-8svbd                                 1/1     Running   0          17h
instana-units      tu-instana-prod-appdata-processor-b6dcc9997-vvqlp          0/1     Pending   0          17h
yaoyao12138 commented 3 years ago

k3s couldn't be used for test 207 but 209.
So 207 only could be tested on kind.