OW 1.1.0 Unable to gathering actions/activation logs on OKD/OpenShift 4.5

[unreal-asr]

Environment: OKD 4.5

Issue Description: I am trying to deploy OW on multi-node OKD 4.5 cluster 3 master node and 3 compute node with support to Dynamic Volume Provision (custom StorageClass)

Other Installation Details: I followed the official guide here

oc label node <only_compute_node_1_2_3> openwhisk-role=invoker

oc adm policy add-scc-to-user anyuid -z default
oc adm policy add-scc-to-user privileged -z default
oc adm policy add-scc-to-user anyuid -z openwhisk-core
oc adm policy add-scc-to-user privileged -z openwhisk-core
oc adm policy add-scc-to-user anyuid -z owdev-init-sa
oc adm policy add-scc-to-user privileged -z owdev-init-sa

Deploying to OKD/OpenShift uses the command sequence:

helm template owdev ./helm/openwhisk -n openwhisk -f _mycluster.yaml_ > owdev.yaml
oc apply -f owdev.yaml  

mycluster.yaml

k8s:
  persistence:
    enabled: true
    hasDefaultStorageClass: false
    explicitStorageClass: ceph-fs

whisk:
  ingress:
    type: OpenShift
    apiHostName: openwhisk.apps.lab.okd.<rest of domain>
    apiHostPort: 443
    apiHostProto: https
    domain: openwhisk.apps.lab.okd.<rest of domain>
  testing:
    includeTests: false

invoker:
  #Tried with and without this option, but no gathering actions/activation logs
  options: "-Dwhisk.spi.LogStoreProvider=org.apache.openwhisk.core.containerpool.logging.LogDriverLogStoreProvider"
  containerFactory:
    impl: kubernetes

I have also tried with:

options: "-Dwhisk.spi.LogStoreProvider=org.apache.openwhisk.core.containerpool.logging.SplunkLogStoreProvider"

Invoker failed with this error:

[2021-01-09T15:42:18.166Z] [ERROR] [#tid_sid_invoker] [Invoker] Failed to initialize reactive invoker: Cannot convert configuration to a org.apache.openwhisk.core.containerpool.logging.SplunkLogStoreConfig. Failures are:
  at 'whisk.logstore':
    - (jar:file:/invoker/lib/openwhisk-common-1.0.0-SNAPSHOT.jar!/application.conf:505) Key not found: 'splunk'.

oc get pods

NAME                                      READY   STATUS    RESTARTS   AGE
owdev-alarmprovider-6f55fd9bd-s9mlz                          1/1     Running     0          13m
owdev-apigateway-5d84c5969d-tg9qx                            1/1     Running     0          13m
owdev-controller-0                                           1/1     Running     0          13m
owdev-couchdb-56885dd9d7-qhwnq                               1/1     Running     0          13m
owdev-invoker-0                                              1/1     Running     0          13m
owdev-kafka-0                                                1/1     Running     0          13m
owdev-kafkaprovider-dfb77c74d-h8xtk                          1/1     Running     0          13m
owdev-redis-7d46bc659-ktnz5                                  1/1     Running     0          13m
owdev-wskadmin                                               1/1     Running     0          13m
owdev-zookeeper-0                                            1/1     Running     0          13m
wskowdev-invoker-00-1-prewarm-nodejs10                       1/1     Running     0          7m57s
wskowdev-invoker-00-2-prewarm-nodejs10                       1/1     Running     0          7m57s
wskowdev-invoker-00-3-whisksystem-invokerhealthtestaction0   1/1     Running     0          7m56s
wskowdev-invoker-00-4-guest-hello-world                      1/1     Running     0          3m32s
wskowdev-invoker-00-5-prewarm-nodejs10                       1/1     Running     0          55s

Issue Detail:

fixed_params.yaml

packages:
  hello_world_package:
    version: 1.0
    license: Apache-2.0
    actions:
      hello_world:
        function: hello.js
        runtime: nodejs:14
        inputs:
          name: Unreal

hello.js

function main(args) {
  const name = args && args.name || "stranger";
  const greeting = `Hello ${name}!`
  console.log(greeting);
  return { greeting: greeting };
}

Deploy

wskdeploy -m fixed_params.yaml

Run Action

wsk -i action invoke /guest/hello_world_package/hello_world 

ok: invoked /guest/hello_world_package/hello_world with id f64a401d53d841f58a401d53d841f540

Verify Activation

wsk activation list

Datetime            Activation ID                    Kind      Start Duration   Status  Entity
2021-01-09 14:36:09 f64a401d53d841f58a401d53d841f540 nodejs:10 cold  90ms       success guest/hello_world:0.0.2

Get Activation by ID

wsk activation get f64a401d53d841f58a401d53d841f540

ok: got activation f64a401d53d841f58a401d53d841f540
{
    "namespace": "guest",
    "name": "hello_world",
    "version": "0.0.2",
    "subject": "guest",
    "activationId": "f64a401d53d841f58a401d53d841f540",
    "start": 1610199369225,
    "end": 1610199369315,
    "duration": 90,
    "statusCode": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "greeting": "Hello Unreal!"
        }
    },
    **"logs": [],**
    "annotations": [
        {
            "key": "path",
            "value": "guest/hello_world_package/hello_world"
        },
        {
            "key": "waitTime",
            "value": 251
        },
        {
            "key": "kind",
            "value": "nodejs:10"
        },
        {
            "key": "timeout",
            "value": false
        },
        {
            "key": "limits",
            "value": {
                "concurrency": 1,
                "logs": 10,
                "memory": 256,
                "timeout": 60000
            }
        },
        {
            "key": "initTime",
            "value": 81
        }
    ],
    "publish": false
}

Verify only logs

wsk activation logs f64a401d53d841f58a401d53d841f540

<no result>

Verify Only result

wsk activation result f64a401d53d841f58a401d53d841f540

{
    "greeting": "Hello Unreal!"
}

Verify invoker pod logs (before it is removed)

oc logs wskowdev-invoker-00-4-guest-hello-world

Hello Unreal!
XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX

I have also read the entire database with wskadmin db get whisks with --docs to include the full documents. No logs entry are present in DB.

Conclusion

Everything seems to work correctly but the logs are not collected with any language (tested also python and java). The exact same test appears to work fine on ow released with docker compose on a single VM It appears to be an ow bug on OKD / OpenShift.

Questions:

Which component of OW is responsible for collecting the invoker / action / activation logs? Useful advice on how to try to solve?

[unreal-asr]

I think the problem is related to CRI-O the new engine container in OpenShift 4.x. CRI-O does not support JSON file logging as described by RedHat here .

I suppose OpenWhisk does not have yet any "logs" implementation for CRI-O text log format.

[paul42]

hey @unreal-altran just circling back around, were you able to fix your issue? Thanks in advance!