Cloud2edge package installation issue - kafka pod initialization problem

[BHUVANESHWARI967]

I'm trying to install Cloud2Edge using the Helm package on my Kubernetes cluster. I get the error as "INSTALLATION FAILED: timed out waiting for the condition". However, the Kafka pod's 'auto-discovery' container fails to start correctly, leading to the pod remaining in a 'Pending' state, and the 'auto-discovery' container enters the 'CrashLoopBackOff' state.

Environment Details: Kubectl Version: v5.0.1 OS: Ubuntu 20.04.6 LTS (Focal Fossa) Docker Version: 20.10.21

Steps Taken:

• Created the required namespace following the instructions from the Cloud2Edge website.
• Attempted to install Cloud2Edge using the provided Helm commands.

Warning BackOff 3m59s (x168 over 51m) kubelet Back-off restarting failed container auto-discovery in pod c2e-kafka-0_lexi(309f640f-f7dc-4790-8b64-967878cf2510)

I attempted to access the container logs using kubectl logs -n abc c2e-kafka-0 -c auto-discovery, but it didn't provide any output. I'm seeking guidance on how to identify the root cause of the issue and how to resolve it. Thank you in advance.

[calohmn]

@BHUVANESHWARI967 Which Kubernetes environment/distribution are you using? In case of minikube, you should run minikube tunnel before installing the chart.

[BHUVANESHWARI967]

@calohmn I have used a single-node kubernetes cluster using kubeadm. Is there way to solve this?

Is only lightweight Kubernetes preferred for installing cloud2edge package?

[calohmn]

The kafka auto-discovery mechanism (as configured in the Hono chart by default) relies on load-balancer support here. If you don't have that in your setup, you can use nodeports. For that, you can adapt the hono.kafka.externalAccess configuration - see here. The configuration to be used is documented in the Kafka chart documentation. You would then also want to set hono.useLoadBalancer to false.

Note that there is currently an issue installing the cloud2edge chart with ditto chart versions greater 3.1.0 (#482). As a workaround you can use the ditto 3.1.0 chart version (see https://github.com/eclipse/packages/issues/482#issuecomment-1672222476) or install your chart based on the changes in #483.

[BHUVANESHWARI967]

Hi, Sorry for the delay in my response. I have the package in minikube cluster. This time the kafka pod is running fine but I am getting timeout error still.

The pod details are,

Can you please help me with the solution of how to solve this?

[calohmn]

Instead of including screenshots, it would be better if you could post the console output as text instead, formatted as a code block. This lets users copy your above helm install command for example and means the text is searchable.

In the helm install command you used above, you are referencing the latest eclipse-iot/cloud2edge chart version from the eclipse-iot packages repo. As noted in my comment above there is currently an issue with that, leading to the issue you encounter.

Note that there is currently an issue installing the cloud2edge chart with ditto chart versions greater 3.1.0 (https://github.com/eclipse/packages/issues/482). As a workaround you can use the ditto 3.1.0 chart version (see https://github.com/eclipse/packages/issues/482#issuecomment-1672222476) or install your chart based on the changes in https://github.com/eclipse/packages/pull/483.

To work around this, you need to checkout the iot-packages repo locally, apply one of the workarounds mentioned above to the local cloud2edge chart file(s) and then install the chart from your local folder. This means, instead of using helm install [...] $RELEASE eclipse-iot/cloud2edge you would use helm install --dependency-update [...] $RELEASE . and run this from the packages/cloud2edge/ directory under your local directory where you cloned the iot-packages repo. The workaround isn't necessary anymore, once #483 has been merged.

[BHUVANESHWARI967]

Thanks for the continuous support. I'll include console output as code from here on.

To work around this, you need to checkout the iot-packages repo locally, apply one of the workarounds mentioned above to the local cloud2edge chart file(s) and then install the chart from your local folder. This means, instead of using helm install [...] $RELEASE eclipse-iot/cloud2edge you would use helm install [...] $RELEASE ./packages/cloud2edge/

Can you tell me what are the charts to be downloaded locally and made changes. And is only the chart.yaml files enough for the installation of c2e package?

[casafurix]

Hello all, I am struggling to even do the basic installation of cloud2edge package as I am constantly facing the timeout exceeded error. I have followed the same steps as in the cloud2edge package website, still it is not getting installed. My device OS is Ubuntu 20.04.

This is my output after running minikube tunnel:

agnibha@leo:~$ minikube tunnel
[sudo] password for agnibha: 
Status: 
    machine: minikube
    pid: 138454
    route: 10.96.0.0/12 -> 192.168.49.2
    minikube: Running
    services: [c2e-adapter-http, c2e-kafka-0-external, my-cloud2edge-adapter-amqp, my-cloud2edge-adapter-http, my-cloud2edge-adapter-mqtt, my-cloud2edge-service-device-registry-ext, eclipse-hono-adapter-amqp, eclipse-hono-adapter-http, eclipse-hono-adapter-mqtt, eclipse-hono-service-device-registry-ext, my-hono-release-2-adapter-amqp, my-hono-release-2-adapter-http, my-hono-release-2-adapter-mqtt, my-hono-release-2-service-device-registry-ext, my-hono-release-3-adapter-amqp, my-hono-release-3-adapter-http, my-hono-release-3-adapter-mqtt, my-hono-release-3-service-device-registry-ext, my-hono-release-4-adapter-http, my-hono-release-4-adapter-mqtt, my-hono-release-4-service-device-registry-ext, my-hono-release-5-adapter-amqp, my-hono-release-5-adapter-http, my-hono-release-5-adapter-mqtt, my-hono-release-5-service-device-registry-ext, my-hono-release-6-adapter-amqp, my-hono-release-6-service-device-registry-ext, my-hono-release-adapter-amqp, my-hono-release-adapter-http, my-hono-release-adapter-mqtt]
    errors: 
        minikube: no errors
        router: no errors
        loadbalancer emulator: no errors

This is the error I get on installation:

agnibha@leo:~$ helm install -n $NS --wait --timeout 30m $RELEASE eclipse-iot/cloud2edge
Error: INSTALLATION FAILED: 3 errors occurred:
    * Timeout: request did not complete within requested timeout - context deadline exceeded
    * Timeout: request did not complete within requested timeout - context deadline exceeded
    * Timeout: request did not complete within requested timeout - context deadline exceeded

Please help, I need help urgently, thanks a lot in advance

[calohmn]

@casafurix There is currently an issue with the cloud2edge chart, caused by an incompatibility with the newest iot-packages ditto chart. Please have a look at my last comment above for a workaround. A new chart version with a fix will be available once #483 has been merged.

EDIT: The cloud2edge chart has been updated now - with the latest chart version 0.4.1 the timeout issues have been resolved. Another thing: If you are using minikube, be sure to run minikube tunnel before installing the chart. (There is a task to update the documentation on this.)

[calohmn]

Can you tell me what are the charts to be downloaded locally and made changes.

@BHUVANESHWARI967 You can download/clone the iot-packages repo, apply the workaround and then use the commands in my earlier comment to install the chart (I've just updated the comment slightly).

Or, you could download this zip file containing the iot-packages repo files with the fix from PR #483 and then change to the packages/cloud2edge/ directory and run helm install --dependency-update [...] $RELEASE . (with your added parameters in place of [...]) from there.

EDIT: The cloud2edge chart has been updated now - with the latest chart version 0.4.1 the timeout issues have been resolved.

And is only the chart.yaml files enough for the installation of c2e package?

A chart consists of all the files in the chart directory, not only Chart.yaml.

[casafurix]

Hello @calohmn , thank you very much for the update.

Whenever I run "minikube tunnel" I get the following output and the process does not terminate.

[sudo] password for agnibha: 
Status:
        machine: minikube
        pid: 343160
        route: 10.96.0.0/12 -> 192.168.49.2
        minikube: Running
        services: []
    errors: 
                minikube: no errors
                router: no errors
                loadbalancer emulator: no errors

So, should I open a new terminal window while this is running and then follow the installation steps, or wait for this to terminate? Thanks in advance, I am actually new to this so I don't have much idea...

[BHUVANESHWARI967]

Hello @casafurix,

If I may answer,

So, should I open a new terminal window while this is running and then follow the installation steps,

Yes, you can start the installation in new terminal alongside the minikube tunnel. You don't have to terminate it I guess.

[casafurix]

The error has stayed for me unfortunately, I am not sure what I am doing wrong here. The following are the steps I followed for installation:

agnibha@leo:~/nthu$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:20:54Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:47:40Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
agnibha@leo:~/nthu$ NS=cloud2edge
agnibha@leo:~/nthu$ kubectl create namespace $NS
namespace/cloud2edge created
agnibha@leo:~/nthu$ RELEASE=c2e
agnibha@leo:~/nthu$ helm install -n $NS --wait --timeout 30m $RELEASE eclipse-iot/cloud2edge
Error: INSTALLATION FAILED: Get "https://192.168.49.2:8443/api/v1/namespaces/cloud2edge/services/c2e-kafka-0-external": context deadline exceeded

And this the output from my minikube tunnel command (I did not terminate it):

[sudo] password for agnibha: 
Status:
        machine: minikube
        pid: 343160
        route: 10.96.0.0/12 -> 192.168.49.2
        minikube: Running
        services: []
    errors: 
                minikube: no errors
                router: no errors
                loadbalancer emulator: no errors
Status:
        machine: minikube
        pid: 343160
        route: 10.96.0.0/12 -> 192.168.49.2
        minikube: Running
        services: [c2e-adapter-amqp, c2e-adapter-http, c2e-adapter-mqtt, c2e-kafka-0-external, c2e-service-device-registry-ext]
    errors: 
                minikube: no errors
                router: no errors
                loadbalancer emulator: no errors

Is there something wrong in these steps? Thanks

[casafurix]

Hello @BHUVANESHWARI967 , If I may ask, did the installation now work for you? I am still facing the same errors as before.

[BHUVANESHWARI967]

Hi @calohmn,

I have attempted to installed the updated chart version by specifying the --version 0.4.1 but it's giving error,

bhu@cloud2edge:~$ helm install -n $NS --wait --timeout 15m --set hono.useLoadBalancer=true --set ditto.nginx.service.type=LoadBalancer $RELEASE eclipse-iot/cloud2edge --version 0.4.1
Error: INSTALLATION FAILED: failed to download "eclipse-iot/cloud2edge" at version "0.4.1"

But then I tried installing without specifying the version of chart,

bhu@cloud2edge:~$ helm install -n $NS --wait --timeout 15m --set hono.useLoadBalancer=true --set ditto.nginx.service.type=LoadBalancer $RELEASE eclipse-iot/cloud2edge
I0821 21:19:52.437606   50111 request.go:601] Waited for 1.03460046s due to client-side throttling, not priority and fairness, request: GET:https://192.168.49.2:8443/api/v1/namespaces/cloud2edge/services/c2e-adapter-amqp
Error: INSTALLATION FAILED: timed out waiting for the condition

bhu@cloud2edge:~$ kubectl get pods -n cloud2edge
NAME                                           READY   STATUS     RESTARTS   AGE
c2e-adapter-amqp-566b57d798-dptv4              0/1     Running    0          17m
c2e-adapter-http-6f5677f6f5-dlqdk              0/1     Running    0          17m
c2e-adapter-mqtt-85dd5c5964-mpj92              0/1     Running    0          17m
c2e-ditto-connectivity-7589d4c5d9-9gvpv        0/1     Running    4          17m
c2e-ditto-gateway-6948479d9f-ndhkg             0/1     Running    3          17m
c2e-ditto-nginx-6d44dd78cb-tc287               0/1     Init:0/1   0          17m
c2e-ditto-policies-58f95899bc-gmdn2            0/1     Running    2          17m
c2e-ditto-swaggerui-578fbb5565-n926f           1/1     Running    0          16m
c2e-ditto-things-585d7f4b55-qq2lt              0/1     Running    4          16m
c2e-ditto-thingssearch-68cf88ff6c-dshp4        0/1     Running    4          17m
c2e-kafka-0                                    0/1     Running    6          17m
c2e-service-auth-57dd95cd44-rvg8h              1/1     Running    0          17m
c2e-service-command-router-f65fb8d58-xngk6     0/1     Running    0          17m
c2e-service-device-registry-6c9cdd8b5d-jdlkm   0/1     Running    0          17m
c2e-zookeeper-0                                0/1     Running    6          17m
ditto-mongodb-69cd845d8-v6zm5                  0/1     Running    7          17m

The pod events taken after installation failed is, c2e_pod_events.txt

Is this an issue or am I doing something wrong here. Please provide me an troubleshooting technique for this.

[casafurix]

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "deployments.apps \"c2e-ditto-connectivity\" is forbidden: User \"system:anonymous\" cannot get resource \"deployments\" in API group \"apps\" in the namespace \"cloud2edge\"",
  "reason": "Forbidden",
  "details": {
    "name": "c2e-ditto-connectivity",
    "group": "apps",
    "kind": "deployments"
  },
  "code": 403
}

When I open the server URL for minikube, this is the json I obtain.. URL: https://192.168.49.2:8443/apis/apps/v1/namespaces/cloud2edge/deployments/c2e-ditto-connectivity

[BHUVANESHWARI967]

Yes, I got that too.

[BHUVANESHWARI967]

@calohmn Can you please take a look into this issue?

[calohmn]

Testing with the helm install parameters you used, my deployment succeeded. First of all, the error

failed to download "eclipse-iot/cloud2edge" at version "0.4.1"

shouldn't be there. Can you run helm repo update eclipse-iot? What is the output of helm search repo eclipse-iot?

The output here

bhu@cloud2edge:~$ kubectl get pods -n cloud2edge
NAME                                           READY   STATUS     RESTARTS   AGE
c2e-adapter-amqp-566b57d798-dptv4              0/1     Running    0          17m
c2e-adapter-http-6f5677f6f5-dlqdk              0/1     Running    0          17m
c2e-adapter-mqtt-85dd5c5964-mpj92              0/1     Running    0          17m
c2e-ditto-connectivity-7589d4c5d9-9gvpv        0/1     Running    4          17m
c2e-ditto-gateway-6948479d9f-ndhkg             0/1     Running    3          17m
...

shows that some of the containers got restarted multiple times.

It would be helpful to find out why. You could run kubectl describe pod --namespace $NS and then look for containers with Last State: Terminated. There is a Reason stated below that line. It could be

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

If that is the case, your containers got killed because not enough memory was available.

How much memory and CPU did you configure for minikube? You can find out by running kubectl get node minikube -o jsonpath='{.status.capacity}'.

These values can be changed by setting the cpus and memory parameters to minikube start. The values --cpus 4 --memory 8192 for 4 CPUs, 8192MiB should be the minimum (see the "Requirements" box on the right of the Cloud2Edge installation page). I'm using 10240 MiB.

[casafurix]

I followed the steps as you mentioned, the following is the output for each command:

agnibha@leo:~$ kubectl get node minikube -o jsonpath='{.status.capacity}'
{"cpu":"8","ephemeral-storage":"238737052Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"24489484Ki","pods":"110"}

agnibha@leo:~$ minikube start --cpus 4 --memory 10240
]😄  minikube v1.31.1 on Ubuntu 20.04
✨  Automatically selected the docker driver. Other choices: qemu2, virtualbox, ssh
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=4, Memory=10240MB) ...
🐳  Preparing Kubernetes v1.27.3 on Docker 24.0.4 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

agnibha@leo:~$ helm repo update eclipse-iot
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "eclipse-iot" chart repository
Update Complete. ⎈Happy Helming!⎈

agnibha@leo:~$ helm search repo eclipse-iot
NAME                        CHART VERSION   APP VERSION     DESCRIPTION                                       
eclipse-iot/cloud2edge      0.4.1           0.4.1           Eclipse IoT Cloud2Edge (C2E) is an integrated s...
eclipse-iot/ditto           3.2.3           3.2.1           DEPRECATED chart of Eclipse Ditto™, which is a ...
eclipse-iot/hawkbit         1.4.2           0.3.0M6-mysql   Eclipse hawkBit™ is a domain independent back-e...
eclipse-iot/hono            2.4.3           2.4.0           Eclipse Hono™ provides remote service interface...
eclipse-iot/telemetry-e2e   0.1.0           0.1.0           The Eclipse IoT Telemetry end-to-end package is...

agnibha@leo:~$ kubectl get pods -n cloud2edge
NAME                                           READY   STATUS     RESTARTS      AGE
c2e-adapter-amqp-849db9f54-dmsjx               0/1     Running    0             9m6s
c2e-adapter-http-54b4f5cdc9-7pf8x              0/1     Running    0             9m10s
c2e-adapter-mqtt-55ffbcf5c8-kh86p              0/1     Running    0             9m6s
c2e-ditto-connectivity-6c47dcf84-ztrrz         1/1     Running    0             9m9s
c2e-ditto-gateway-86f7b57f5c-zrxgq             1/1     Running    0             9m10s
c2e-ditto-nginx-7f77857745-qfrm2               1/1     Running    0             9m6s
c2e-ditto-policies-6546d74968-bp62c            1/1     Running    0             9m6s
c2e-ditto-swaggerui-64fb576fcf-qpx5h           1/1     Running    0             9m9s
c2e-ditto-things-7b55c79998-fhlz4              1/1     Running    0             9m10s
c2e-ditto-thingssearch-6f4c4d866d-rb4hn        1/1     Running    0             9m8s
c2e-kafka-0                                    0/1     Init:0/1   3 (88s ago)   9m9s
c2e-service-auth-5cc4c68d66-s49nr              1/1     Running    0             9m9s
c2e-service-command-router-6f4fdc5ff4-jqqnk    0/1     Running    0             9m6s
c2e-service-device-registry-5b985cfc86-cnd9q   0/1     Running    0             9m10s
c2e-zookeeper-0                                1/1     Running    0             9m9s
ditto-mongodb-5457498f87-wxbsg                 1/1     Running    0             9m10s

Even after following all steps, this is my installation command's output:

agnibha@leo:~$ helm install -n $NS --wait --timeout 15m --set hono.useLoadBalancer=true --set ditto.nginx.service.type=LoadBalancer $RELEASE eclipse-iot/cloud2edge
Error: INSTALLATION FAILED: etcdserver: request timed out

Can you figure out the error from these or is there something else I should check apart from these?

I did run kubectl describe pod --namespace $NS as well but I am unable to understand the issue from its output. This is my text file with this command's output: c2e_pod_events_agnibha.txt

Thanks.

[BHUVANESHWARI967]

@calohmn

Can you run helm repo update eclipse-iot? What is the output of helm search repo eclipse-iot?

I did run these two commands and the output was,

bhu@cloud2edge:~$ helm repo update eclipse-iot
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "eclipse-iot" chart repository
Update Complete. ⎈Happy Helming!⎈

bhu@cloud2edge:~$ helm search repo eclipse-iot
NAME                        CHART VERSION   APP VERSION     DESCRIPTION                                       
eclipse-iot/cloud2edge      0.4.1           0.4.1           Eclipse IoT Cloud2Edge (C2E) is an integrated s...
eclipse-iot/ditto           3.2.3           3.2.1           DEPRECATED chart of Eclipse Ditto™, which is a ...
eclipse-iot/hawkbit         1.4.2           0.3.0M6-mysql   Eclipse hawkBit™ is a domain independent back-e...
eclipse-iot/hono            2.4.3           2.4.0           Eclipse Hono™ provides remote service interface...
eclipse-iot/telemetry-e2e   0.1.0           0.1.0           The Eclipse IoT Telemetry end-to-end package is...

Then I tried installing again with minikube tunnel running alongside,

bhu@cloud2edge:~$ helm install -n $NS --wait --timeout 15m --set hono.useLoadBalancer=true --set ditto.nginx.service.type=LoadBalancer $RELEASE eclipse-iot/cloud2edge --version 0.4.1 --generate-name
I0822 09:55:35.955239   14510 request.go:601] Waited for 1.007370889s due to client-side throttling, not priority and fairness, request: GET:https://192.168.49.2:8443/api/v1/namespaces/cloud2edge/services/cloud2edge-1692677465-adapter-http
Error: INSTALLATION FAILED: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline

$kubectl describe pod --namespace $NS for this I got, c2e_pod_describe.txt

I have created minikube cluster with enough space as mentioned in doc,

bhu@cloud2edge:~$ kubectl get node minikube -o jsonpath='{.status.capacity}'
{"cpu":"4","ephemeral-storage":"1055243732Ki","hugepages-2Mi":"0","memory":"10977904Ki","pods":"110"}

Can you provide an suggestion for this?

Also can you answer this question in discussion, https://github.com/eclipse-ditto/ditto/discussions/1719

[calohmn]

@casafurix The init container auto-discovery of the Kafka pod (c2e-kafka-0) didn't get completed in your case.

Name:             c2e-kafka-0
[...]
Init Containers:
  auto-discovery:
[...]
    Image:         docker.io/bitnami/kubectl:1.25.6-debian-11-r11
[...]
    Command:
      /scripts/auto-discovery.sh
    State:          Running
      Started:      Tue, 22 Aug 2023 12:42:17 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
[...]
    Ready:          False
    Restart Count:  6
[...]
Back-off restarting failed container auto-discovery in pod c2e-kafka-0_cloud2edge

Could you check the logs of this container? Just to be sure: Did you have minikube tunnel running in a separate terminal?

The describe pods output you provided also showed that quite some time was spent pulling the images. When you try again and the images are already available locally, startup should be faster.

---

@BHUVANESHWARI967 In your case the mongodb container doesn't start (and that's the reason the other pods don't get ready):

Containers:
  mongodb:
[...]
    Image:          docker.io/bitnami/mongodb:5.0.10-debian-11-r3
[...]
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    132

Error code 132 seems to indicate that your CPU isn't supported by the used MongoDB version - see https://forums.docker.com/t/docker-container-exited-with-code-132/113173 .

[calohmn]

@BHUVANESHWARI967 Workaround for the MongoDB exit code 132 error could be to use an older MongoDB image of version 4. This requires some adaptation to the liveness/readiness probe commands then. You could try adding these parameters to helm install:

--set ditto.mongodb.image.tag=4
--set "ditto.mongodb.customLivenessProbe.exec.command={pgrep,mongod}"
--set "ditto.mongodb.customReadinessProbe.exec.command={mongo,--eval,db.adminCommand('ping')}"

Or you put these values in a separate yaml file containing

ditto:
  mongodb:
    image:
      tag: 4
    customLivenessProbe:
      exec:
        command:
          - pgrep
          - mongod
    customReadinessProbe:
      exec:
        command:
          - mongo
          - --eval
          - "db.adminCommand('ping')"

and reference that for helm install (with -f).

[BHUVANESHWARI967]

@calohmn, Thanks for continuous support!

I tried the helm install command,

bhu@cloud2edge:~$ helm install -n $NS --wait --timeout 15m --set hono.useLoadBalancer=true --set ditto.nginx.service.type=LoadBalancer --set ditto.mongodb.image.tag=4 --set "ditto.mongodb.customLivenessProbe.exec.command={pgrep,mongod}" --set "ditto.mongodb.customReadinessProbe.exec.command={mongod,--eval,db.adminCommand('ping')}" $RELEASE eclipse-iot/cloud2edge --version 0.4.1
E0823 16:10:54.097646   16549 request.go:977] Unexpected error when reading response body: context deadline exceeded
Error: INSTALLATION FAILED: unexpected error when reading response body. Please retry. Original error: context deadline exceeded

bhu@cloud2edge:~$ kubectl get pods -n cloud2edge
NAME                                           READY   STATUS              RESTARTS   AGE
c2e-adapter-amqp-85f4c7d54f-d2kdh              0/1     Running             0          17m
c2e-adapter-http-6fc9b9b58d-vk7ms              0/1     Running             0          17m
c2e-adapter-mqtt-5588895bc5-hmwzt              0/1     Running             0          17m
c2e-ditto-connectivity-56d88977c5-z8ljm        0/1     Running             4          17m
c2e-ditto-gateway-7cdb766b4d-9jgtg             0/1     Running             4          17m
c2e-ditto-nginx-69f7f5fd9d-zwvvt               0/1     Init:0/1            0          17m
c2e-ditto-policies-56dff694d5-n7x5q            0/1     Running             4          17m
c2e-ditto-swaggerui-cf7785996-fl6kg            1/1     Running             0          17m
c2e-ditto-things-857d568f-mpsz4                0/1     Running             4          17m
c2e-ditto-thingssearch-7657cbf98f-c6fq8        0/1     Running             4          17m
c2e-kafka-0                                    0/1     Running             6          17m
c2e-service-auth-b4d79cf4f-l8mf9               1/1     Running             0          17m
c2e-service-command-router-7bf669f84c-bkdr2    0/1     Running             0          17m
c2e-service-device-registry-67dc989bff-wwbwm   0/1     Running             0          17m
c2e-zookeeper-0                                1/1     Running             5          17m
ditto-mongodb-6ddc8b4cfd-9wk94                 0/1     ContainerCreating   0          17m

The pod desc for this is, c2e_pod_describe2.txt

Then I have tried the other approach by creating a yaml file with the data you have provided, (ran minikube tunnel, included the repos and updated the repo)

bhu@cloud2edge:~$ helm install -n $NS --wait --timeout 15m --set hono.useLoadBalancer=true --set ditto.nginx.service.type=LoadBalancer -f ditto.yaml $RELEASE eclipse-iot/cloud2edge --version 0.4.1
Error: INSTALLATION FAILED: client rate limiter Wait returned an error: context deadline exceeded

bhu@cloud2edge:~$ kubectl get pods -n cloud2edge
NAME                                           READY   STATUS             RESTARTS   AGE
c2e-adapter-amqp-85f4c7d54f-t89f9              0/1     Running            0          15m
c2e-adapter-http-6fc9b9b58d-fmlv8              0/1     Running            0          15m
c2e-adapter-mqtt-5588895bc5-2dht5              0/1     Running            0          15m
c2e-ditto-connectivity-56d88977c5-mg9gz        1/1     Running            1          15m
c2e-ditto-gateway-dfd58ff59-t6x6c              1/1     Running            0          15m
c2e-ditto-nginx-7dbd67b64-4pfbc                1/1     Running            0          15m
c2e-ditto-policies-56dff694d5-kqs4w            1/1     Running            0          15m
c2e-ditto-swaggerui-cf7785996-nr6pf            1/1     Running            0          15m
c2e-ditto-things-857d568f-b897h                1/1     Running            0          15m
c2e-ditto-thingssearch-7657cbf98f-284jw        1/1     Running            0          15m
c2e-kafka-0                                    0/1     CrashLoopBackOff   7          15m
c2e-service-auth-b4d79cf4f-nxqsg               1/1     Running            0          15m
c2e-service-command-router-7bf669f84c-7hm9h    0/1     Running            0          15m
c2e-service-device-registry-67dc989bff-j9jwb   0/1     Running            0          15m
c2e-zookeeper-0                                1/1     Running            2          15m
ditto-mongodb-747f9856bb-mjknh                 1/1     Running            0          15m

Don't know why the kafka pod is failing, kafka_pod_desc.txt

[BHUVANESHWARI967]

@casafurix Hi, Did you find a way to solve this?

[calohmn]

@BHUVANESHWARI967 Looking into the kafka_pod_desc.txt you provided above, there is this:

Containers:
  kafka:
    Container ID:  docker://77ec8b8b0a9e73e095793aabc098c8edbc9ef570bdca515b4633e04c96773ff4
    Image:         docker.io/bitnami/kafka:3.4.0-debian-11-r4
[...]
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137

Googling for this exit code 137 reveals that this means the process got terminated because it was using too much memory.

In the kubectl get node minikube output you posted above

bhu@cloud2edge:~$ kubectl get node minikube -o jsonpath='{.status.capacity}'
{"cpu":"4","ephemeral-storage":"1055243732Ki","hugepages-2Mi":"0","memory":"10977904Ki","pods":"110"}

it shows that you have 10GB of memory available all in all. If you have configured minikube with 8GB, this doesn't leave much room for the OS and other applications. You could have a look at the overall memory usage in your system, when running helm install. Concerning memory usage in Kubernetes, you could enable the minikube metrics-server via minikube addons enable metrics-server and then run kubectl top pods --all-namespaces to get info about the memory usage of the pods.

[BHUVANESHWARI967]

@calohmn Thank you for your patience with my issue.

That issue seems to persist even after increasing the storage of minikube cluster. The pod description also highlighted a connection timed out with probe failing.

Is there any stable version for the cloud2edge package that I can install? If so, can you please give the steps to do so?

[casafurix]

@casafurix Hi, Did you find a way to solve this?

It hasn't worked for me either, have been trying this, facing issues in this too. I think there is some issue in the package...

https://github.com/ertis-research/OpenTwins#deploying-eclipse-ditto-and-eclipse-hono

[calohmn]

That issue seems to persist even after increasing the storage of minikube cluster.

@BHUVANESHWARI967 To me it seems the issue is with the physical amount of RAM in your machine - and possibly with the amount of memory the OS and other applications are taking up. Have you monitored the memory usage, e.g. via Task Manager if you are using Windows? Do you still get Exit Code: 137 errors in the kubectl describe output for the Kafka pod?

[casafurix]

@calohmn can we use MicroK8s instead of minikube? Actually I tried the MicroK8s installation steps, even that seems to have errors in the installation steps. So do we only have minikube as option?

[BHUVANESHWARI967]

@calohmn Thank you, I got it to work by increasing the disk space.

[casafurix]

@calohmn Thank you, I got it to work by increasing the disk space.

Hello @BHUVANESHWARI967 , can you please let me know what exactly you changed which worked for you? I did assign 10 GB memory (total memory 24 GB) and I have 105 GB free disk space, so I do not that is the issue for me. Can you tell me the steps which you might have changed? Thanks

[calohmn]

@casafurix You could also try MicroK8s. But, if you already have Minikube running, I would suggest rather to try finding out why the installation in Minikube failed. The Kafka auto-discovery pod errors shown in your comment above is something I have only seen if minikube tunnel isn't running, so you should make sure you have it running. Before running helm install you could check kubectl get pods --all-namespaces to check if there are remaining cloud2edge pods in "Terminating" state from a previous installation attempt. The latest minikube version seems to have an issue cleaning up those (at least on my system, didn't have the issue with minikube v1.29.0). Stopping and starting minikube again removes these pods. You could also increase the timeout parameter for the helm install command to, let's say, 30m: helm install -n $NS --wait --timeout 30m --set hono.useLoadBalancer=true --set ditto.nginx.service.type=LoadBalancer $RELEASE eclipse-iot/cloud2edge. If you deleted the minikube cluster before and now are running the chart deployment for the first time, downloading the docker images could take quite a long time, therefore better to use a higher value here. If the installation still fails, you could again check the "kubectl describe" output and check the logs of the pods/containers that failed/didn't get ready.

[casafurix]

Hello @calohmn,

For some reason yesterday, when I added this extra argument for disk size, minikube start --driver=docker --cpus 4 --memory 10240 --disk-size=41g

After which I ran the default helm install command as mentioned on the website with a timeout of 15s, and it worked! Thank you so much for your continuous help!

This is the output for kubectl get pods --all-namespaces now:

agnibha@leo:~$ kubectl get pods --all-namespaces
NAMESPACE              NAME                                           READY   STATUS    RESTARTS       AGE
cloud2edge             c2e-adapter-amqp-849db9f54-l6hss               1/1     Running   0              19h
cloud2edge             c2e-adapter-http-54b4f5cdc9-s75kj              1/1     Running   0              19h
cloud2edge             c2e-adapter-mqtt-55ffbcf5c8-gxbbn              1/1     Running   0              19h
cloud2edge             c2e-ditto-connectivity-6c47dcf84-zvx5w         1/1     Running   0              19h
cloud2edge             c2e-ditto-gateway-954cbdd99-vggkc              1/1     Running   0              19h
cloud2edge             c2e-ditto-nginx-dd757f4cf-mtspn                1/1     Running   0              19h
cloud2edge             c2e-ditto-policies-6546d74968-rjqtc            1/1     Running   0              19h
cloud2edge             c2e-ditto-swaggerui-64fb576fcf-fnb4x           1/1     Running   0              19h
cloud2edge             c2e-ditto-things-7b55c79998-zqq29              1/1     Running   0              19h
cloud2edge             c2e-ditto-thingssearch-6f4c4d866d-xfgc6        1/1     Running   0              19h
cloud2edge             c2e-kafka-0                                    1/1     Running   2 (162m ago)   19h
cloud2edge             c2e-service-auth-5cc4c68d66-r7mvf              1/1     Running   0              19h
cloud2edge             c2e-service-command-router-6f4fdc5ff4-lsktx    1/1     Running   0              19h
cloud2edge             c2e-service-device-registry-5b985cfc86-6drqt   1/1     Running   0              19h
cloud2edge             c2e-zookeeper-0                                1/1     Running   0              19h
cloud2edge             ditto-mongodb-5457498f87-8s7c9                 1/1     Running   0              19h
kube-system            coredns-5d78c9869d-c5hvf                       1/1     Running   0              19h
kube-system            etcd-minikube                                  1/1     Running   0              19h
kube-system            kube-apiserver-minikube                        1/1     Running   0              19h
kube-system            kube-controller-manager-minikube               1/1     Running   0              19h
kube-system            kube-proxy-wd6zc                               1/1     Running   0              19h
kube-system            kube-scheduler-minikube                        1/1     Running   0              19h
kube-system            storage-provisioner                            1/1     Running   1 (19h ago)    19h
kubernetes-dashboard   dashboard-metrics-scraper-5cb4f4bb9c-mf5wq     1/1     Running   0              19h
kubernetes-dashboard   kubernetes-dashboard-6967859bff-g7mk2          1/1     Running   0              19h

[calohmn]

@casafurix That's great! One further note here: The cloud2edge chart was updated yesterday - you should make sure you are using version 0.4.2 (checking with helm search repo eclipse-iot and running helm repo update eclipse-iot if you don't use the new version yet). The tour (and the setCloud2EdgeEnv.sh script mentioned there) has also been updated.

[casafurix]

Actually in the tour, when I run the following

agnibha@leo:~$ curl -i -k -u demo-device@org.eclipse.packages.c2e:demo-secret -H 'application/json' --data-binary '{
>   "topic": "org.eclipse.packages.c2e/demo-device/things/twin/commands/modify",
>   "headers": {},
>   "path": "/features/temperature/properties/value",
>   "value": 45
> }' ${HTTP_ADAPTER_BASE_URL:?}/telemetry

I always am getting this error

HTTP/1.1 401 Unauthorized
content-type: application/json; charset=utf-8
content-length: 27
{"error":"bad credentials"}

Not sure how to add credentials here, I have set up the setCloud2EdgeEnv.sh script as well. I have the updated the charts as well, with respect to your previous comment:

agnibha@leo:~$ helm search repo eclipse-iot
NAME                        CHART VERSION   APP VERSION     DESCRIPTION                                       
eclipse-iot/cloud2edge      0.4.1           0.4.1           Eclipse IoT Cloud2Edge (C2E) is an integrated s...
eclipse-iot/ditto           3.2.3           3.2.1           DEPRECATED chart of Eclipse Ditto™, which is a ...
eclipse-iot/hawkbit         1.4.2           0.3.0M6-mysql   Eclipse hawkBit™ is a domain independent back-e...
eclipse-iot/hono            2.4.3           2.4.0           Eclipse Hono™ provides remote service interface...
eclipse-iot/telemetry-e2e   0.1.0           0.1.0           The Eclipse IoT Telemetry end-to-end package is...
agnibha@leo:~$ helm repo update eclipse-iot
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "eclipse-iot" chart repository
Update Complete. ⎈Happy Helming!⎈
agnibha@leo:~$ helm search repo eclipse-iot
NAME                        CHART VERSION   APP VERSION     DESCRIPTION                                       
eclipse-iot/cloud2edge      0.4.2           0.4.2           Eclipse IoT Cloud2Edge (C2E) is an integrated s...
eclipse-iot/ditto           3.2.3           3.2.1           DEPRECATED chart of Eclipse Ditto™, which is a ...
eclipse-iot/hawkbit         1.4.2           0.3.0M6-mysql   Eclipse hawkBit™ is a domain independent back-e...
eclipse-iot/hono            2.4.3           2.4.0           Eclipse Hono™ provides remote service interface...
eclipse-iot/telemetry-e2e   0.1.0           0.1.0           The Eclipse IoT Telemetry end-to-end package is...

[calohmn]

@casafurix The credentials should have been added automatically via a chart post-install hook. Have you done a helm uninstall and then again helm install of the cloud2edge chart after having done the helm repo update eclipse-iot? (Also check there are no remaining cloud2edge pods in "terminating" state afterhelm uninstall after a minute or so - restart the minikube cluster if that's the case.)

[casafurix]

That worked...thank you so much for your constant help! Really grateful!