apache / openwhisk-deploy-kube

The Apache OpenWhisk Kubernetes Deployment repository supports deploying the Apache OpenWhisk system on Kubernetes and OpenShift clusters.
https://openwhisk.apache.org/
Apache License 2.0
296 stars 228 forks source link

Controller status is either running or CrashLoopBackOff and invoker waiting to initialize #722

Open harsh61426 opened 2 years ago

harsh61426 commented 2 years ago

Im unable to get the Controller and Invoker to running 1/1 state.

$kubectl get pods -n openwhisk
NAME                                   READY   STATUS      RESTARTS   AGE
owdev-alarmprovider-6c89d8846b-rjpqf   1/1     Running     0          95d
owdev-apigateway-d6d756db8-x7kb4       1/1     Running     0          95d
owdev-controller-0                     0/1     Running     15         3h29m
owdev-couchdb-5666bfc5f6-gnktm         1/1     Running     0          95d
owdev-gen-certs-649zd                  0/1     Completed   0          95d
owdev-init-couchdb-f5g6k               0/1     Completed   0          95d
owdev-install-packages-7psb4           0/1     Completed   0          95d
owdev-invoker-0                        0/1     Init:0/1    0          4h1m
owdev-kafka-0                          1/1     Running     0          4h10m
owdev-kafkaprovider-7867778f74-rhhzt   1/1     Running     0          95d
owdev-nginx-67b8974c77-dcwt4           1/1     Running     0          95d
owdev-redis-cdb68fc8f-tdczh            1/1     Running     0          95d
owdev-wskadmin                         1/1     Running     0          95d
owdev-zookeeper-0                      1/1     Running     0          4h10m

$ kubectl logs owdev-controller-0 -n openwhisk
[2022-02-14T06:53:43.612Z] [INFO] Slf4jLogger started
[2022-02-14T06:53:44.040Z] [WARN] Failed to attach the instrumentation because the Kamon Bundle is not present on the classpath
[2022-02-14T06:53:44.196Z] [INFO] Started the Kamon StatsD reporter
[2022-02-14T06:53:44.504Z] [INFO] [#tid_sid_unknown] [Config] environment set value for limits.triggers.fires.perMinute
[2022-02-14T06:53:44.505Z] [INFO] [#tid_sid_unknown] [Config] environment set value for limits.actions.sequence.maxLength
[2022-02-14T06:53:44.505Z] [INFO] [#tid_sid_unknown] [Config] environment set value for limits.actions.invokes.concurrent
[2022-02-14T06:53:44.505Z] [INFO] [#tid_sid_unknown] [Config] environment set value for limits.actions.invokes.perMinute
[2022-02-14T06:53:44.506Z] [INFO] [#tid_sid_unknown] [Config] environment set value for runtimes.manifest
[2022-02-14T06:53:44.506Z] [INFO] [#tid_sid_unknown] [Config] environment set value for kafka.hosts
[2022-02-14T06:53:44.506Z] [INFO] [#tid_sid_unknown] [Config] environment set value for port
[2022-02-14T06:53:45.186Z] [INFO] [#tid_sid_unknown] [KafkaMessagingProvider] completed0 already exists and the user can see it, skipping creation.
[2022-02-14T06:53:45.366Z] [INFO] [#tid_sid_unknown] [KafkaMessagingProvider] health already exists and the user can see it, skipping creation.
[2022-02-14T06:53:45.534Z] [INFO] [#tid_sid_unknown] [KafkaMessagingProvider] cacheInvalidation already exists and the user can see it, skipping creation.
[2022-02-14T06:53:45.678Z] [INFO] [#tid_sid_unknown] [KafkaMessagingProvider] events already exists and the user can see it, skipping creation.
[2022-02-14T06:53:45.871Z] [INFO] [#tid_sid_controller] [Controller] starting controller instance 0 [marker:controller_startup0_counter:1393]
[2022-02-14T06:53:46.527Z] [INFO] [#tid_sid_dispatcher] [MessageFeed] handler capacity = 128, pipeline fill at = 128, pipeline depth = 256
[2022-02-14T06:53:46.718Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] managedFraction = 0.9, blackboxFraction = 0.1
[2022-02-14T06:53:46.782Z] [INFO] [#tid_sid_dispatcher] [MessageFeed] handler capacity = 128, pipeline fill at = 128, pipeline depth = 256
[2022-02-14T06:53:47.131Z] [INFO] [#tid_sid_loadbalancer] [WhiskAction] [GET] serving from datastore: CacheKey(whisk.system/invokerHealthTestAction0) [marker:database_cacheMiss_counter:2654]
[2022-02-14T06:53:47.183Z] [INFO] [#tid_sid_loadbalancer] [CouchDbRestStore] [GET] 'test_whisks' finding document: 'id: whisk.system/invokerHealthTestAction0' [marker:database_getDocument_start:2706]
[2022-02-14T06:53:47.376Z] [INFO] [#tid_sid_loadbalancer] [CouchDbRestStore]  [marker:database_getDocument_finish:2899:192]
[2022-02-14T06:53:47.427Z] [INFO] [#tid_sid_loadbalancer] [WhiskAction] invalidating CacheKey(whisk.system/invokerHealthTestAction0)
Exception in thread "main" org.apache.openwhisk.core.database.DocumentTypeMismatchException: document type class org.apache.openwhisk.core.entity.WhiskTrigger did not match expected type class org.apache.openwhisk.core.entity.WhiskAction.
[2022-02-14T06:53:47.432Z] [ERROR] [#tid_sid_loadbalancer] [InvokerPool] error creating test action for invoker health: org.apache.openwhisk.core.database.DocumentTypeMismatchException: document type class org.apache.openwhisk.core.entity.WhiskTrigger did not match expected type class org.apache.openwhisk.core.entity.WhiskAction.
    at org.apache.openwhisk.core.database.StoreUtils$.deserialize(StoreUtils.scala:71)
    at org.apache.openwhisk.core.database.CouchDbRestStore.$anonfun$get$5(CouchDbRestStore.scala:233)
    at org.apache.openwhisk.core.database.CouchDbRestStore$$Lambda$1297/0000000058052500.apply(Unknown Source)
    at scala.util.Success.$anonfun$map$1(Try.scala:255)
    at scala.util.Success.map(Try.scala:213)
    at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
    at scala.concurrent.Future$$Lambda$658/00000000A0007890.apply(Unknown Source)
    at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
    at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
    at scala.concurrent.impl.Promise$$Lambda$164/00000000091939C0.apply(Unknown Source)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
    at akka.dispatch.BatchingExecutor$BlockableBatch$$Lambda$1131/000000007C0031D0.apply$mcV$sp(Unknown Source)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:836)

$ kubectl describe pod owdev-controller-0 -n openwhisk
Name:         owdev-controller-0
Namespace:    openwhisk
Priority:     0
Node:         dvorak-2-3/10.1.0.23
Start Time:   Sun, 13 Feb 2022 21:36:04 -0600
Labels:       app=owdev-openwhisk
              chart=openwhisk-1.0.1
              controller-revision-hash=owdev-controller-c6c445889
              heritage=Helm
              name=owdev-controller
              release=owdev
              statefulset.kubernetes.io/pod-name=owdev-controller-0
Annotations:  <none>
Status:       Running
IP:           10.244.1.227
IPs:
  IP:           10.244.1.227
Controlled By:  StatefulSet/owdev-controller
Init Containers:
  wait-for-kafka:
    Container ID:  docker://6227eb6e184643a13a14256caf5f9a3c1c05f11c5300a4cc329013cc86772326
    Image:         openwhisk/ow-utils:3e6138d
    Image ID:      docker-pullable://openwhisk/ow-utils@sha256:0b1a3531ade6774069f3f5777784f1d2446d2165537dd617bbc7302bcfb690b1
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      cacert="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"; token="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"; while true; do rc=$(curl -sS --cacert $cacert --header "Authorization: Bearer $token" https://kubernetes.default.svc/api/v1/namespaces/openwhisk/endpoints/owdev-kafka | jq -r ".subsets[].addresses | length"); echo "num ready kafka endpoints is $rc"; if [ $rc -gt 0 ]; then echo "Success: ready kafka endpoint!"; break; fi; echo "kafka not ready yet; sleeping for 3 seconds"; sleep 3; done;
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 13 Feb 2022 21:36:06 -0600
      Finished:     Sun, 13 Feb 2022 21:36:06 -0600
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from owdev-core-token-4ptj9 (ro)
  wait-for-couchdb:
    Container ID:  docker://eb91335d1e63d2eeb74bb37d3509f026fb2157d76e896a4c1bfb0e46ba1ec0d3
    Image:         busybox:latest
    Image ID:      docker-pullable://busybox@sha256:139abcf41943b8bcd4bc5c42ee71ddc9402c7ad69ad9e177b0a9bc4541f14924
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      while true; do echo 'checking CouchDB readiness'; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; if [ $result -eq 0 ]; then echo 'Success: CouchDB is ready!'; break; fi; echo '...not ready yet; sleeping 3 seconds before retry'; sleep 3; done;
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 13 Feb 2022 21:36:07 -0600
      Finished:     Sun, 13 Feb 2022 21:36:07 -0600
    Ready:          True
    Restart Count:  0
    Environment:
      READINESS_URL:  http://owdev-couchdb.openwhisk.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from owdev-core-token-4ptj9 (ro)
Containers:
  controller:
    Container ID:  docker://50bf9291af60ceaaa609157a376819b13469cc17ed3b71681033f2651ef8c51d
    Image:         openwhisk/controller:3e6138d
    Image ID:      docker-pullable://openwhisk/controller@sha256:51e0f338a942fe395358263cf5f76f673bb949b3d36494556fac760a22df899a
    Ports:         8080/TCP, 2552/TCP, 19999/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Command:
      /bin/bash
      -c
      /init.sh `hostname | awk -F '-' '{print $NF}'`
    State:          Running
      Started:      Mon, 14 Feb 2022 00:53:41 -0600
    Last State:     Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Mon, 14 Feb 2022 00:40:21 -0600
      Finished:     Mon, 14 Feb 2022 00:53:40 -0600
    Ready:          False
    Restart Count:  15
    Liveness:       http-get http://:8080/ping delay=200s timeout=600s period=200s #success=1 #failure=3
    Readiness:      http-get http://:8080/ping delay=200s timeout=600s period=200s #success=1 #failure=3
    Environment:
      PORT:                                                        8080
      TZ:                                                          UTC
      CONFIG_whisk_info_date:                                      <set to the key 'whisk_info_date' of config map 'owdev-whisk.config'>     Optional: false
      CONFIG_whisk_info_buildNo:                                   <set to the key 'whisk_info_buildNo' of config map 'owdev-whisk.config'>  Optional: false
      JAVA_OPTS:                                                   -Xmx1024M 
      CONTROLLER_OPTS:                                              
      RUNTIMES_MANIFEST:                                           {
                                                                       "runtimes": {
                                                                           "nodejs": [
                                                                               {
                                                                                   "kind": "nodejs:12",
                                                                                   "default": false,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-nodejs-v12",
                                                                                       "tag": "1.19.0"
                                                                                   },
                                                                                   "deprecated": false,
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   }
                                                                               },
                                                                               {
                                                                                   "kind": "nodejs:14",
                                                                                   "default": true,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-nodejs-v14",
                                                                                       "tag": "1.19.0"
                                                                                   },
                                                                                   "deprecated": false,
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   },
                                                                                   "stemCells": [
                                                                                       {
                                                                                           "initialCount": 2,
                                                                                           "memory": "256 MB",
                                                                                           "reactive": {
                                                                                               "minCount": 1,
                                                                                               "maxCount": 4,
                                                                                               "ttl": "2 minutes",
                                                                                               "threshold": 1,
                                                                                               "increment": 1
                                                                                           }
                                                                                       }
                                                                                   ]
                                                                               }
                                                                           ],
                                                                           "python": [
                                                                               {
                                                                                   "kind": "python:3",
                                                                                   "default": true,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-python-v3.7",
                                                                                       "tag": "1.17.0"
                                                                                   },
                                                                                   "deprecated": false,
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   }
                                                                               }
                                                                           ],
                                                                           "swift": [
                                                                               {
                                                                                   "kind": "swift:4.2",
                                                                                   "default": true,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-swift-v4.2",
                                                                                       "tag": "1.17.0"
                                                                                   },
                                                                                   "deprecated": false,
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   }
                                                                               },
                                                                               {
                                                                                   "kind": "swift:5.1",
                                                                                   "default": false,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-swift-v5.1",
                                                                                       "tag": "1.17.0"
                                                                                   },
                                                                                   "deprecated": false,
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   }
                                                                               },
                                                                               {
                                                                                   "kind": "swift:5.3",
                                                                                   "default": false,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-swift-v5.3",
                                                                                       "tag": "1.17.0"
                                                                                   },
                                                                                   "deprecated": false,
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   }
                                                                               },
                                                                               {
                                                                                   "kind": "swift:5.4",
                                                                                   "default": false,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-swift-v5.4",
                                                                                       "tag": "1.17.0"
                                                                                   },
                                                                                   "deprecated": false,
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   }
                                                                               }
                                                                           ],
                                                                           "java": [
                                                                               {
                                                                                   "kind": "java:8",
                                                                                   "default": true,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "java8action",
                                                                                       "tag": "1.17.0"
                                                                                   },
                                                                                   "deprecated": false,
                                                                                   "attached": {
                                                                                       "attachmentName": "jarfile",
                                                                                       "attachmentType": "application/java-archive"
                                                                                   },
                                                                                   "requireMain": true
                                                                               }
                                                                           ],
                                                                           "php": [
                                                                               {
                                                                                   "kind": "php:7.3",
                                                                                   "default": false,
                                                                                   "deprecated": false,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-php-v7.3",
                                                                                       "tag": "1.17.0"
                                                                                   },
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   }
                                                                               },
                                                                               {
                                                                                   "kind": "php:7.4",
                                                                                   "default": true,
                                                                                   "deprecated": false,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-php-v7.4",
                                                                                       "tag": "1.17.0"
                                                                                   },
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   }
                                                                               },
                                                                               {
                                                                                   "kind": "php:8.0",
                                                                                   "default": false,
                                                                                   "deprecated": false,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-php-v7.4",
                                                                                       "tag": "1.17.0"
                                                                                   },
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   }
                                                                               }
                                                                           ],
                                                                           "ruby": [
                                                                               {
                                                                                   "kind": "ruby:2.5",
                                                                                   "default": true,
                                                                                   "deprecated": false,
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   },
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-ruby-v2.5",
                                                                                       "tag": "1.17.0"
                                                                                   }
                                                                               }
                                                                           ],
                                                                           "go": [
                                                                               {
                                                                                   "kind": "go:1.15",
                                                                                   "default": true,
                                                                                   "deprecated": false,
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   },
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-golang-v1.15",
                                                                                       "tag": "1.18.0"
                                                                                   }
                                                                               }
                                                                           ],
                                                                           "rust": [
                                                                               {
                                                                                   "kind": "rust:1.34",
                                                                                   "default": true,
                                                                                   "deprecated": false,
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   },
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-rust-v1.34",
                                                                                       "tag": "1.3.0"
                                                                                   }
                                                                               }
                                                                           ],
                                                                           "dotnet": [
                                                                               {
                                                                                   "kind": "dotnet:2.2",
                                                                                   "default": true,
                                                                                   "deprecated": false,
                                                                                   "requireMain": true,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-dotnet-v2.2",
                                                                                       "tag": "1.16.0"
                                                                                   },
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   }
                                                                               },
                                                                               {
                                                                                   "kind": "dotnet:3.1",
                                                                                   "default": false,
                                                                                   "deprecated": false,
                                                                                   "requireMain": true,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-dotnet-v3.1",
                                                                                       "tag": "1.16.0"
                                                                                   },
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   }
                                                                               }
                                                                           ],
                                                                           "ballerina": [
                                                                               {
                                                                                   "kind": "ballerina:0.990",
                                                                                   "default": true,
                                                                                   "image": {
                                                                                       "prefix": "openwhisk",
                                                                                       "name": "action-ballerina-v0.990.2",
                                                                                       "tag": "nightly"
                                                                                   },
                                                                                   "deprecated": false,
                                                                                   "attached": {
                                                                                       "attachmentName": "codefile",
                                                                                       "attachmentType": "text/plain"
                                                                                   }
                                                                               }
                                                                           ]
                                                                       },
                                                                       "blackboxes": [
                                                                           {
                                                                               "prefix": "openwhisk",
                                                                               "name": "dockerskeleton",
                                                                               "tag": "1.14.0"
                                                                           }
                                                                       ]
                                                                   }

      LIMITS_ACTIONS_INVOKES_PERMINUTE:                            60
      LIMITS_ACTIONS_INVOKES_CONCURRENT:                           30
      LIMITS_TRIGGERS_FIRES_PERMINUTE:                             60
      LIMITS_ACTIONS_SEQUENCE_MAXLENGTH:                           50
      CONFIG_whisk_timeLimit_min:                                  100ms
      CONFIG_whisk_timeLimit_max:                                  5m
      CONFIG_whisk_timeLimit_std:                                  1m
      CONFIG_whisk_memory_min:                                     128m
      CONFIG_whisk_memory_max:                                     512m
      CONFIG_whisk_memory_std:                                     256m
      CONFIG_whisk_concurrencyLimit_min:                           1
      CONFIG_whisk_concurrencyLimit_max:                           1
      CONFIG_whisk_concurrencyLimit_std:                           1
      CONFIG_whisk_logLimit_min:                                   0m
      CONFIG_whisk_logLimit_max:                                   10m
      CONFIG_whisk_logLimit_std:                                   10m
      CONFIG_whisk_activation_payload_max:                         1048576
      CONFIG_whisk_loadbalancer_blackboxFraction:                  10%
      CONFIG_whisk_loadbalancer_timeoutFactor:                     2
      KAFKA_HOSTS:                                                 owdev-kafka-0.owdev-kafka.openwhisk.svc.cluster.local:9092
      CONFIG_whisk_kafka_replicationFactor:                        
      CONFIG_whisk_kafka_topics_cacheInvalidation_retentionBytes:  
      CONFIG_whisk_kafka_topics_cacheInvalidation_retentionMs:     
      CONFIG_whisk_kafka_topics_cacheInvalidation_segmentBytes:    
      CONFIG_whisk_kafka_topics_completed_retentionBytes:          
      CONFIG_whisk_kafka_topics_completed_retentionMs:             
      CONFIG_whisk_kafka_topics_completed_segmentBytes:            
      CONFIG_whisk_kafka_topics_events_retentionBytes:             
      CONFIG_whisk_kafka_topics_events_retentionMs:                
      CONFIG_whisk_kafka_topics_events_segmentBytes:               
      CONFIG_whisk_kafka_topics_health_retentionBytes:             
      CONFIG_whisk_kafka_topics_health_retentionMs:                
      CONFIG_whisk_kafka_topics_health_segmentBytes:               
      CONFIG_whisk_kafka_topics_invoker_retentionBytes:            
      CONFIG_whisk_kafka_topics_invoker_retentionMs:               
      CONFIG_whisk_kafka_topics_invoker_segmentBytes:              
      CONFIG_whisk_couchdb_username:                               <set to the key 'db_username' in secret 'owdev-db.auth'>        Optional: false
      CONFIG_whisk_couchdb_password:                               <set to the key 'db_password' in secret 'owdev-db.auth'>        Optional: false
      CONFIG_whisk_couchdb_port:                                   <set to the key 'db_port' of config map 'owdev-db.config'>      Optional: false
      CONFIG_whisk_couchdb_protocol:                               <set to the key 'db_protocol' of config map 'owdev-db.config'>  Optional: false
      CONFIG_whisk_couchdb_host:                                   owdev-couchdb.openwhisk.svc.cluster.local
      CONFIG_whisk_couchdb_provider:                               <set to the key 'db_provider' of config map 'owdev-db.config'>           Optional: false
      CONFIG_whisk_couchdb_databases_WhiskActivation:              <set to the key 'db_whisk_activations' of config map 'owdev-db.config'>  Optional: false
      CONFIG_whisk_couchdb_databases_WhiskEntity:                  <set to the key 'db_whisk_actions' of config map 'owdev-db.config'>      Optional: false
      CONFIG_whisk_couchdb_databases_WhiskAuth:                    <set to the key 'db_whisk_auths' of config map 'owdev-db.config'>        Optional: false
      CONTROLLER_INSTANCES:                                        1
      CONFIG_logback_log_level:                                    INFO
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from owdev-core-token-4ptj9 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  owdev-core-token-4ptj9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  owdev-core-token-4ptj9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                     From     Message
  ----     ------     ----                    ----     -------
  Warning  Unhealthy  8m15s (x45 over 3h24m)  kubelet  Readiness probe failed: Get "http://10.244.1.227:8080/ping": dial tcp 10.244.1.227:8080: connect: connection refused
  Warning  Unhealthy  2m10s (x47 over 3h25m)  kubelet  Liveness probe failed: Get "http://10.244.1.227:8080/ping": dial tcp 10.244.1.227:8080: connect: connection refused

I increased the initialDelaySeconds for the liveness and readiness probe for the controller, but it doesn't work. Can you please tell me how to resolve this issue?

style95 commented 2 years ago

I formatted your description for better readability. It seems it complains about invokerHealthTestAction. How did you initialize CouchDB?

harsh61426 commented 2 years ago

Thanks for formatting. To initialize CouchDB, I had initially tried to run this script https://github.com/apache/openwhisk-deploy-kube/blob/master/helm/openwhisk/configMapFiles/initCouchDB/initdb.sh But it didn't work. So I downloaded the relevant files from the OpenWhisk repo and used the CouchDB parameters given in values.yaml, ran the ansible playbook commands.

Note: everything was working fine for the past few weeks

Is there a way to check invokerHealthTestAction?

style95 commented 2 years ago

I would look into the DB.

It generally looks like this.

{
  "_id": "whisk.system/invokerHealthTestAction0",
  "_rev": "187-cb37241d627e51fc27ba74690ab13fd2",
  "annotations": [],
  "entityType": "action",
  "exec": {
    "kind": "nodejs:10",
    "code": "function main(params) { return params; }",
    "binary": false
  },
  "limits": {
    "concurrency": 1,
    "logs": 0,
    "memory": 128,
    "timeout": 60000
  },
  "name": "invokerHealthTestAction0",
  "namespace": "whisk.system",
  "parameters": [],
  "publish": false,
  "updated": 1641283227482,
  "version": "0.0.1"
}
harsh61426 commented 2 years ago

I restarted the pods using kubectl rollout restart deploy which restarted alarmprovider, apigateway, couched, kafkaprovider, nginx and redis. After this the controller and invoker are now in running state.

But I have encountered other problems with OpenWhisk. The whisk namespace is empty when I check " wsk -i property get". This is not allowing action creation and invocations. I can't seem to find a way to set this value. Hence I tried reinstalling the couchDB part with no success.

The documentation part on deploying and initializing the couchDB is not clear. How exactly do I do it? Do I run all the scripts in https://github.com/apache/openwhisk-deploy-kube/tree/master/helm/openwhisk/configMapFiles ? In what order should I run them?

dgrove-oss commented 2 years ago

The scripts are run by the owdev-init-couchdb job when the helm chart is first deployed. They are not intended to be run except during this initial deployment.