Closed sgloutnikov closed 1 year ago
Hello, I am actually unable to install the cloud2edge package using the steps mentioned on the website (facing many errors), can you please guide me the right steps now to install it? I want to use Ditto+Hono. Thanks
(Cloud2edge installation issues were resolved with @casafurix in #481. In general, the Gitter channel or Stackoverflow (with eclipse-iot tag) are good places for asking questions, or, in case of a chart or documentation bug, an issue can be raised here.)
@sgloutnikov yes, the cloud2edge chart should be updated. A PR is welcome here.
@sgloutnikov The update actually looked quite straightforward. I've integrated it in PR #493 .
Really strange, I'm unable to bring up the environment in a fresh k3s or minikube single node cluster, any tips are greatly appreciated. This behaviour seems identical for k3s with LoadBalancer or minikube without.
Looks like a bunch of Ditto pods failing with rediness probe and Akka errors in logs, which chains down to nginx and swagger-ui not starting up.
Helm install:
cloud2edge git:(pr/493) helm install -n $NS --wait --timeout 20m --set hono.useLoadBalancer=true --set ditto.nginx.service.type=LoadBalancer $RELEASE ./
Get pods:
➜ ~ k get pods
NAME READY STATUS RESTARTS AGE
c2e-ditto-nginx-6859974f89-5s5gv 0/1 Init:0/1 0 10m
c2e-ditto-dittoui-649d4b487d-jz7nt 1/1 Running 0 10m
c2e-ditto-gateway-9fc695954-tmb75 0/1 Running 0 10m
c2e-ditto-thingssearch-76c969667d-mpmqk 0/1 Running 0 10m
c2e-ditto-policies-56fc485445-pxmqb 0/1 Running 0 10m
c2e-ditto-things-6944d6d6fd-rzjn7 0/1 Running 0 10m
c2e-ditto-connectivity-7d94b56b85-85rzt 0/1 Running 0 10m
c2e-zookeeper-0 1/1 Running 0 10m
ditto-mongodb-778c99d5bf-2lsq5 1/1 Running 0 10m
c2e-kafka-0 1/1 Running 0 10m
c2e-service-auth-7d7d8dfdf6-p4d84 1/1 Running 0 10m
c2e-service-device-registry-86484b7f7b-wkjxg 1/1 Running 0 10m
c2e-service-command-router-587d5c68fd-kg5fl 1/1 Running 0 10m
c2e-adapter-mqtt-687c66bbdd-548pc 1/1 Running 0 10m
c2e-adapter-amqp-77579d66f4-hgzt8 1/1 Running 0 10m
c2e-adapter-http-d56575cbf-kmlkl 1/1 Running 0 10m
c2e-ditto-swaggerui-646678bdfd-5lkr8 0/1 Init:CrashLoopBackOff 6 (3m36s ago) 10m
Get services:
➜ ~ k get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
c2e-kafka-headless ClusterIP None <none> 9092/TCP,9093/TCP 10m
c2e-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 10m
ditto-mongodb ClusterIP 10.43.206.8 <none> 27017/TCP 10m
c2e-service-device-registry ClusterIP 10.43.89.217 <none> 5671/TCP,8080/TCP,8443/TCP 10m
c2e-service-auth ClusterIP 10.43.125.245 <none> 5671/TCP,8088/TCP 10m
c2e-ditto-swaggerui ClusterIP 10.43.22.151 <none> 8080/TCP 10m
c2e-service-command-router ClusterIP 10.43.133.236 <none> 5671/TCP 10m
c2e-ditto-dittoui ClusterIP 10.43.239.86 <none> 8080/TCP 10m
c2e-ditto-gateway ClusterIP 10.43.142.65 <none> 8080/TCP 10m
c2e-kafka ClusterIP 10.43.32.223 <none> 9092/TCP,9094/TCP 10m
c2e-zookeeper ClusterIP 10.43.159.167 <none> 2181/TCP,2888/TCP,3888/TCP 10m
c2e-kafka-0-external LoadBalancer 10.43.151.26 172.XX.XX.XX 9094:32094/TCP 10m
c2e-adapter-amqp LoadBalancer 10.43.163.121 172.XX.XX.XX 5671:31200/TCP 10m
c2e-ditto-nginx LoadBalancer 10.43.193.12 172.XX.XX.XX 8080:30341/TCP 10m
c2e-service-device-registry-ext LoadBalancer 10.43.181.30 172.XX.XX.XX 28443:31396/TCP 10m
c2e-adapter-http LoadBalancer 10.43.50.24 172.XX.XX.XX 8443:32470/TCP 10m
c2e-adapter-mqtt LoadBalancer 10.43.186.5 172.XX.XX.XX 8883:32334/TCP 10m
Describe nginx pod and logs:
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
...
Normal Pulling 12m kubelet Pulling image "rancher/curlimages-curl:7.73.0"
Normal Pulled 12m kubelet Successfully pulled image "rancher/curlimages-curl:7.73.0" in 3.785419237s (3.785428409s including waiting)
Normal Created 12m kubelet Created container wait-for-gateway
Normal Started 12m kubelet Started container wait-for-gateway
~ k logs c2e-ditto-nginx-6859974f89-5s5gv
Defaulted container "ditto-nginx" out of: ditto-nginx, wait-for-gateway (init)
Error from server (BadRequest): container "ditto-nginx" in pod "c2e-ditto-nginx-6859974f89-5s5gv" is waiting to start: PodInitializing
Describe gateway pod:
Containers:
ditto-gateway:
Container ID: containerd://bcc2029e8d57fa99a7cc8514f6ef97eb9cc743131ad7bb9feedd0db9bd5d57db
Image: docker.io/eclipse/ditto-gateway:3.3.6
...
Liveness: http-get http://:management/alive delay=160s timeout=5s period=10s #success=1 #failure=4
Readiness: http-get http://:management/ready delay=30s timeout=3s period=10s #success=1 #failure=3
Things Logs:
k logs c2e-ditto-things-6944d6d6fd-rzjn7
...
{"@timestamp":"2023-09-07T22:19:03.654+02:00","@version":"1","message":"Probing [http://10-42-0-21.spg-dev.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Exceeded configured max-open-requests value of [32]. This means that the request queue of this pool (HostConnectionPoolSetup(10-42-0-21.spg-dev.pod.cluster.local,8558,ConnectionPoolSetup(ConnectionPoolSettings(4,0,0,32,1,Duration.Inf,100 milliseconds,2 minutes,30 seconds,Duration.Inf,ClientConnectionSettings(Some(User-Agent: akka-http/10.2.10),10 seconds,60 seconds,512,None,WebSocketSettings(<function0>,ping,Duration.Inf,akka.http.impl.settings.WebSocketSettingsImpl$$$Lambda$1168/0x00007f1aeb6e0738@4c674487,false),List(),ParserSettings(2048,16,64,64,8192,64,Some(9223372036854775807),8388608,256,1048576,5,Strict,RFC6265,true,Set(),Full,Error,Error,Error,HashMap(If-Range -> 0, If-Modified-Since -> 0, If-Unmodified-Since -> 0, default -> 12, If-None-Match -> 0, User-Agent -> 32, Content-MD5 -> 0, Date -> 0, If-Match -> 0),false,false,true,akka.util.ConstantFun$$$Lambda$1116/0x00007f1aeb6a7248@35e995d0,akka.util.ConstantFun$$$Lambda$1116/0x00007f1aeb6a7248@35e995d0,akka.util.ConstantFun$$$Lambda$1117/0x00007f1aeb6a7610@3c4dff45),100 milliseconds,None,Http2ClientSettingsImpl(256,65536,10000000,512000,1024,false,0 seconds,0 seconds,0,3 seconds,100 milliseconds,2 minutes,None),TCPTransport),1 second,List()),akka.http.scaladsl.HttpConnectionContext$@5dc42dd8,akka.event.MarkerLoggingAdapter@6c563e20))) has completely filled up because the pool currently does not process requests fast enough to handle the incoming request load. Please retry the request later. See https://doc.akka.io/docs/akka-http/current/scala/http/client-side/pool-overflow.html for more information.","logger_name":"akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap","thread_name":"ditto-cluster-akka.actor.default-dispatcher-11","level":"WARN","level_value":30000,"akkaAddress":"akka://ditto-cluster@10.42.0.18:2551","sourceThread":"ditto-cluster-akka.actor.default-dispatcher-11","akkaSource":"akka://ditto-cluster@10.42.0.18:2551/system/bootstrapCoordinator/contactPointProbe-10-42-0-21.spg-dev.pod.cluster.local-8558"}
{"@timestamp":"2023-09-07T22:19:03.935+02:00","@version":"1","message":"Discovered [5] contact points, confirmed [0], which is less than the required [5], retrying","logger_name":"akka.management.cluster.bootstrap.LowestAddressJoinDecider","thread_name":"ditto-cluster-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000,"akkaAddress":"akka://ditto-cluster@10.42.0.18:2551","akkaContactPoints":"10-42-0-21.spg-dev.pod.cluster.local:8558, 10-42-0-22.spg-dev.pod.cluster.local:8558, 10-42-0-17.spg-dev.pod.cluster.local:8558, 10-42-0-18.spg-dev.pod.cluster.local:8558, 10-42-0-19.spg-dev.pod.cluster.local:8558","sourceThread":"ditto-cluster-akka.actor.default-dispatcher-13","akkaSource":"LowestAddressJoinDecider(akka://ditto-cluster)","akkaSeedNodes":"","tags":["akkaBootstrapInProgress"]}
{"@timestamp":"2023-09-07T22:19:04.204+02:00","@version":"1","message":"Probing [http://10-42-0-17.spg-dev.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Exceeded configured max-open-requests value of [32]. This means that the request queue of this pool (HostConnectionPoolSetup(10-42-0-17.spg-dev.pod.cluster.local,8558,ConnectionPoolSetup(ConnectionPoolSettings(4,0,0,32,1,Duration.Inf,100 milliseconds,2 minutes,30 seconds,Duration.Inf,ClientConnectionSettings(Some(User-Agent: akka-http/10.2.10),10 seconds,60 seconds,512,None,WebSocketSettings(<function0>,ping,Duration.Inf,akka.http.impl.settings.WebSocketSettingsImpl$$$Lambda$1168/0x00007f1aeb6e0738@4c674487,false),List(),ParserSettings(2048,16,64,64,8192,64,Some(9223372036854775807),8388608,256,1048576,5,Strict,RFC6265,true,Set(),Full,Error,Error,Error,HashMap(If-Range -> 0, If-Modified-Since -> 0, If-Unmodified-Since -> 0, default -> 12, If-None-Match -> 0, User-Agent -> 32, Content-MD5 -> 0, Date -> 0, If-Match -> 0),false,false,true,akka.util.ConstantFun$$$Lambda$1116/0x00007f1aeb6a7248@35e995d0,akka.util.ConstantFun$$$Lambda$1116/0x00007f1aeb6a7248@35e995d0,akka.util.ConstantFun$$$Lambda$1117/0x00007f1aeb6a7610@3c4dff45),100 milliseconds,None,Http2ClientSettingsImpl(256,65536,10000000,512000,1024,false,0 seconds,0 seconds,0,3 seconds,100 milliseconds,2 minutes,None),TCPTransport),1 second,List()),akka.http.scaladsl.HttpConnectionContext$@5dc42dd8,akka.event.MarkerLoggingAdapter@6c563e20))) has completely filled up because the pool currently does not process requests fast enough to handle the incoming request load. Please retry the request later. See https://doc.akka.io/docs/akka-http/current/scala/http/client-side/pool-overflow.html for more information.","logger_name":"akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap","thread_name":"ditto-cluster-akka.actor.default-dispatcher-12","level":"WARN","level_value":30000,"akkaAddress":"akka://ditto-cluster@10.42.0.18:2551","sourceThread":"ditto-cluster-akka.actor.default-dispatcher-13","akkaSource":"akka://ditto-cluster@10.42.0.18:2551/system/bootstrapCoordinator/contactPointProbe-10-42-0-17.spg-dev.pod.cluster.local-8558"}
@sgloutnikov Good point. I actually wasn't testing with ditto.nginx.service.type=LoadBalancer
. I was testing without any such chart parameters being set (the hono.useLoadBalancer=true
is the default anyway - we need to adapt the documentation there).
About ditto.nginx.service.type=LoadBalancer
we need to check whether there's an issue there with the Ditto chart. I actually don't see setting this option to LoadBalancer being documented in the Ditto repo.
(EDIT: I've tested using ditto.nginx.service.type=LoadBalancer
now and found no issues.)
@calohmn If I install each chart separately, the Hono chart installed fine. However, the latest Ditto chart has the same problem as above. I am installing with the default command which pulls the latest version: helm install -n ditto my-ditto oci://registry-1.docker.io/eclipse/ditto --wait
.
In what environment are you testing the cloud2edge chart that it's able to install properly? I'm unable to get it to install your PR version, with or without extra parameters, on my local single node k3s or minikube.
@sgloutnikov I'm using minikube v1.29.0 (with v1.31.2 I have the issue that pods are stuck in "terminating" state when uninstalling the chart). I'm running minikube tunnel
.
I'm convinced the installation issues above are specific to my environment, most likely due to DNS inside the Kubernetes cluster and pods being unable to reach each other for the Akka readiness checks. I was able to install the Ditto Hono chart in minikube with a VirtualBox driver when the Docker one before was giving me the above-mentioned problems. Can't pin it down exactly, but will report if I find more.
Resolved in #493
I just noticed that the Ditto helm chart used in cloud2edge requirements.yaml is no longer maintained in this repository but was moved to Ditto's repository and is also served from a different Helm repository. Does this mean that the cloud2edge package should be updated to use the new and updated Ditto Helm chart, or are there some incompatibilities that could be preventing the switch over?
I'll test on my end and see how far I can get with updating the cloud2edge Ditto version to latest.