"waiting for the Whisk controller to come up ..." forever

[aleks-myplanet]

Hi, I'm getting this exact same issue since Mon Dec 9th 2019 (may have started sooner but was fine on Fri Dec 6th 2019). Was there any updates to the docker images that perhaps broken it once again?

Originally posted by @aleks-myplanet in https://github.com/apache/openwhisk-devtools/issues/219#issuecomment-564629023

[style95]

@aleks-myplanet Could you share more about your environment(mac, linux, etc) to reproduce this?

[style95]

ok I tried to reproduce this on mac and found this error:

Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.apache.openwhisk.common.Logging.info$default$3(Logging.scala:54)
        at org.apache.openwhisk.common.Logging.info$default$3$(Logging.scala:54)
        at org.apache.openwhisk.common.AkkaLogging.info$default$3(Logging.scala:92)
        at org.apache.openwhisk.common.Config$.$anonfun$readPropertiesFromSystemAndEnv$1(Config.scala:149)
        at scala.collection.mutable.HashMap$$anon$1.$anonfun$foreach$2(HashMap.scala:153)
        at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
        at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
        at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:153)
        at org.apache.openwhisk.common.Config$.readPropertiesFromSystemAndEnv(Config.scala:145)
        at org.apache.openwhisk.common.Config.getProperties(Config.scala:119)
        at org.apache.openwhisk.core.WhiskConfig.getProperties(WhiskConfig.scala:51)
        at org.apache.openwhisk.common.Config.<init>(Config.scala:48)
        at org.apache.openwhisk.core.WhiskConfig.<init>(WhiskConfig.scala:43)
        at org.apache.openwhisk.core.controller.Controller$.start(Controller.scala:227)
        at org.apache.openwhisk.core.controller.Controller$.main(Controller.scala:213)
        at org.apache.openwhisk.core.controller.Controller.main(Controller.scala)
Caused by: pureconfig.error.ConfigReaderException: Cannot convert configuration to a org.apache.openwhisk.common.MetricConfig. Failures are:
  at 'whisk.metrics.kamon-enabled':
    - Expected type BOOLEAN. Found STRING instead.
  at 'whisk.metrics.logs-enabled':
    - Expected type BOOLEAN. Found STRING instead.

        at pureconfig.package$.getResultOrThrow(package.scala:139)
        at pureconfig.package$.loadConfigOrThrow(package.scala:161)
        at org.apache.openwhisk.common.TransactionId$.<init>(TransactionId.scala:207)
        at org.apache.openwhisk.common.TransactionId$.<clinit>(TransactionId.scala)

[rabbah]

We'll need to add the parameter to the config. Thanks @style95 for investigating this.

[leonardas103]

I restarted wsk (make stop && make start) and now I am also getting this error. I did make destroy then make quick-start and the error still persists. I also noted a new error:

/bin/sh: 28: let: not found`

Tired it on two machines (ubuntu 18.04 and 19.04).
Also, unrelated but the make destroy throws this error:

...
Removing openwhisk_minio_1           ... done
cleaning other OpenWhisk containers started by the invoker ... 
"docker stop" requires at least 1 argument.
See 'docker stop --help'.

Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]

Stop one or more running containers
"docker rm" requires at least 1 argument.
See 'docker rm --help'.

Usage:  docker rm [OPTIONS] CONTAINER [CONTAINER...]

[aleks-myplanet]

Getting everything above that @leonardas103 posted. Tried it on multiple machines, Ubuntu 18.04, Mac OS Mojave.

[rabbah]

Hi @aleks-myplanet - we have a regression here because of a new parameter requirement for the controller. We need to path the deployment to provide this value whisk.metrics.kamon-enabled: false to the controller.

[arcsud91]

@rabbah Is this issue resolved, or is there a change in the way we run this quick-start?

[style95]

@arcsud91 This issue is not resolved yet. I tried to fix it but seems not that simple.

I could find the required environment variables in a container or some configurations in container args. But it shows such errors.

For example, whisk.metrics.kamon-enabled is defined in application.conf file and the value can be overridden like this:

        kamon-enabled      = ${?METRICS_KAMON}

METRIC_KAMON is also defined in the container args.

I confirmed those values are the same as the native setup. But still, it complains about the type.

I could avoid such errors by adding these configurations in docker-compose.yml, but it keeps complaining about other configurations as well.

      METRICS_KAMON: "False"
      METRICS_LOG: "True"

I suspect there is a fundamental issue in populating such configurations.

So I was trying to figure out the difference between docker-compose and native setup. After then, I could not have enough time to work on, I decided to come up to this later again.

Any contribution would be very appreciated.

[style95]

@arcsud91 I figured it out and #309 would fix this. In the meantime, you can try with lower-case values.

[4n70w4]

The same error now in current master.

waiting for the Whisk controller to come up ...
.............................................................................................................................

[upd] okay, make destroy then make quick-start fix it.