Update Max memory and time OpenWhisk Backend - Fails Actions updated to use the new values

[rads18]

Environment details:

*Kubernetes, kubernetes 1.13

Docker version 18.09.7

Steps to reproduce the issue:

1. Changed the default max for memory and time as below: actions: time: min: "100ms"

   LT: Updated

   max: "5m"

   max: "15m"

   LT

   std: "1m" memory: min: "128m"

   LT: Updated

   max: "512m"

   max: "1024m"

   LT

   std: "256m"

2. Post change create or update any function (python3) to use new max Like change the function to use "-m 768" or '-m 1024" . The function fails to work and errors out with response "response": { "status": "whisk internal error", "statusCode": 0, "success": false, "result": { "error": "Action version is not compatible and cannot be invoked." } }
3. However, updating the function back to the old max still works.
4. There are no errors in updating function to the new max memory number only when invoking the function it fails.
5. invoker was restarted as per my knowledge but no difference.

Provide the expected results and outputs:

The function should be transparent to this memory update

Provide the actual results and outputs:

output comes here

Additional information you deem important:

• The issue is persistent once the action is updated to use any value above the previous max 512 Also as i pointed above the issue is fixed when the function is updated back to use any value under or equal to the old max 512

[style95]

How did you increase the limits? Did you increase the system limits as well?

[rads18]

yes Openwhisk was configured to the new limits if thats what you mean?

[rads18]

hello team, Can someone provide me insight if there is something we am doing wrong or this falls in as a bug? If it is possible to increase the default system limits on action I'd appreciate if you could share a link on steps (assuming it is documented) so we can investigate what we did wrong.

thank you!

[style95]

Since in our downstream, we are running OpenWhisk with higher memory and time limits, I am suspicious the way you configured new limits.

Could you share the details on how did you configure them?

[rads18]

Hi style

The team member that admins that environment is OOO at the moment if you can share which file do you expect these changes to be made?

(the content of the file i have shared in my initial post) thanks

[style95]

Those configurations conform to this file: https://github.com/apache/openwhisk/blob/master/common/scala/src/main/resources/application.conf#L437

And I think you can configure them with these ansible variables. https://github.com/apache/openwhisk/blob/05ed4e18eb53f5630735ffbe28726dce5a3481b5/ansible/roles/controller/tasks/deploy.yml#L212 https://github.com/apache/openwhisk/blob/05ed4e18eb53f5630735ffbe28726dce5a3481b5/ansible/roles/controller/tasks/deploy.yml#L216

[rads18]

Thank you style for your inputs, i will be testing these with the admin in next day or so after they return. i found the ansible variable quite interesting , are these being set for a play called by OpenWhisk(OW) admin privileges ? Is there a way for me to pass when calling the action from ansible? or action parameters at whisk?

Note: I am not the admin for OW but have complete access in my namepsace on OW.

[rads18]

Hi Style,

I checked with my admin and they did update the controller file and also see the new values when they describe the pod on openwhisk-controller.

I have attached configuration file with new values. We basically took the max to 1MB.

Please let me know your thoughts thanks

( i dont see a way to attach so including it in here) The file name is openwhisk-controller_0.yaml

`apiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/psp: aaaa-vault-psp creationTimestamp: "2020-09-03T20:06:54Z" generateName: openwhisk-controller- labels: app: openwhisk-openwhisk chart: openwhisk-0.1.6 controller-revision-hash: openwhisk-controller-6f6587bfb8 heritage: Tiller name: openwhisk-controller release: openwhisk statefulset.kubernetes.io/pod-name: openwhisk-controller-0 name: openwhisk-controller-0 namespace: openwhisk ownerReferences:

• apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: StatefulSet name: openwhisk-controller uid: d9e57809-def6-11e9-868d-0050568a8ca4 resourceVersion: "1214198130" selfLink: /api/v1/namespaces/openwhisk/pods/openwhisk-controller-0 uid: 035b6989-ee21-11ea-bc9f-0050568a790a spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution:
  • preference: matchExpressions:
    • key: openwhisk-role operator: NotIn values:
      • invoker weight: 100
  • preference: matchExpressions:
    • key: openwhisk-role operator: In values:
      • core weight: 80 podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution:
  • labelSelector: matchExpressions:
    • key: name operator: In values:
      • openwhisk-controller topologyKey: kubernetes.io/hostname containers:
• command:
  • /bin/bash
  • -c
  • /init.sh hostname | cut -d'-' -f3 env:
  • name: PORT value: "8080"
  • name: TZ value: UTC
  • name: CONFIG_whisk_info_date valueFrom: configMapKeyRef: key: whisk_info_date name: openwhisk-whisk.config
  • name: CONFIG_whisk_info_buildNo valueFrom: configMapKeyRef: key: whisk_info_buildNo name: openwhisk-whisk.config
  • name: JAVA_OPTS value: '-Xmx1024M '
  • name: CONTROLLER_OPTS value: ' '
  • name: RUNTIMES_MANIFEST value: | { "runtimes": { "nodejs": [ { "kind": "nodejs:6", "default": false, "image": { "prefix": "openwhisk", "name": "nodejs6action", "tag": "1.14.0-incubating" }, "deprecated": false, "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" } }, { "kind": "nodejs:8", "default": false, "image": { "prefix": "openwhisk", "name": "action-nodejs-v8", "tag": "1.14.0-incubating" }, "deprecated": false, "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" } }, { "kind": "nodejs:10", "default": true, "image": { "prefix": "openwhisk", "name": "action-nodejs-v10", "tag": "1.14.0-incubating" }, "deprecated": false, "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" }, "stemCells": [ { "count": 2, "memory": "256 MB" } ] }, { "kind": "nodejs:12", "default": false, "image": { "prefix": "openwhisk", "name": "action-nodejs-v12", "tag": "1.14.0-incubating" }, "deprecated": false, "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" } } ], "python": [ { "kind": "python:2", "default": false, "image": { "prefix": "openwhisk", "name": "python2action", "tag": "1.13.0-incubating" }, "deprecated": false, "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" } }, { "kind": "python:3", "default": true, "image": { "prefix": "openwhisk", "name": "python3action", "tag": "1.13.0-incubating" }, "deprecated": false, "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" } } ], "swift": [ { "kind": "swift:4.2", "default": true, "image": { "prefix": "openwhisk", "name": "action-swift-v4.2", "tag": "1.13.0-incubating" }, "deprecated": false, "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" } } ], "java": [ { "kind": "java:8", "default": true, "image": { "prefix": "openwhisk", "name": "java8action", "tag": "1.13.0-incubating" }, "deprecated": false, "attached": { "attachmentName": "jarfile", "attachmentType": "application/java-archive" }, "requireMain": true } ], "php": [ { "kind": "php:7.3", "default": true, "deprecated": false, "image": { "prefix": "openwhisk", "name": "action-php-v7.3", "tag": "1.13.0-incubating" }, "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" } } ], "ruby": [ { "kind": "ruby:2.5", "default": true, "deprecated": false, "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" }, "image": { "prefix": "openwhisk", "name": "action-ruby-v2.5", "tag": "1.13.0-incubating" } } ], "go": [ { "kind": "go:1.11", "default": true, "deprecated": false, "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" }, "image": { "prefix": "openwhisk", "name": "actionloop-golang-v1.11", "tag": "1.13.0-incubating" } } ], "dotnet": [ { "kind": "dotnet:2.2", "default": true, "deprecated": false, "requireMain": true, "image": { "prefix": "openwhisk", "name": "action-dotnet-v2.2", "tag": "1.13.0-incubating" }, "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" } } ], "ballerina": [ { "kind": "ballerina:0.990", "default": true, "image": { "prefix": "openwhisk", "name": "action-ballerina-v0.990.2", "tag": "nightly" }, "deprecated": false, "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" } } ] }, "blackboxes": [ { "prefix": "openwhisk", "name": "dockerskeleton", "tag": "1.13.0-incubating" } ] }
  • name: LIMITS_ACTIONS_INVOKES_PERMINUTE value: "60"
  • name: LIMITS_ACTIONS_INVOKES_CONCURRENT value: "30"
  • name: LIMITS_TRIGGERS_FIRES_PERMINUTE value: "60"
  • name: LIMITS_ACTIONS_SEQUENCE_MAXLENGTH value: "50"
  • name: CONFIG_whisk_timeLimit_min value: 100ms
  • name: CONFIG_whisk_timeLimit_max value: 15m
  • name: CONFIG_whisk_timeLimit_std value: 1m
  • name: CONFIG_whisk_memory_min value: 128m - name: CONFIG_whisk_memory_max value: 1024m
  • name: CONFIG_whisk_memory_std value: 256m
  • name: CONFIG_whisk_concurrencyLimit_min value: "1"
  • name: CONFIG_whisk_concurrencyLimit_max value: "1"
  • name: CONFIG_whisk_concurrencyLimit_std value: "1"
  • name: CONFIG_whisk_logLimit_min value: 0m
  • name: CONFIG_whisk_logLimit_max value: 10m
  • name: CONFIG_whisk_logLimit_std value: 10m
  • name: CONFIG_whisk_activation_payload_max value: "1048576"
  • name: CONFIG_whisk_loadbalancer_blackboxFraction value: 10%
  • name: CONFIG_whisk_loadbalancer_timeoutFactor value: "2"
  • name: KAFKA_HOSTS value: openwhisk-kafka-0.openwhisk-kafka.openwhisk.svc.cluster.local:9092
  • name: CONFIG_whisk_kafka_replicationFactor
  • name: CONFIG_whisk_kafka_topics_cacheInvalidation_retentionBytes
  • name: CONFIG_whisk_kafka_topics_cacheInvalidation_retentionMs
  • name: CONFIG_whisk_kafka_topics_cacheInvalidation_segmentBytes
  • name: CONFIG_whisk_kafka_topics_completed_retentionBytes
  • name: CONFIG_whisk_kafka_topics_completed_retentionMs
  • name: CONFIG_whisk_kafka_topics_completed_segmentBytes
  • name: CONFIG_whisk_kafka_topics_events_retentionBytes
  • name: CONFIG_whisk_kafka_topics_events_retentionMs
  • name: CONFIG_whisk_kafka_topics_events_segmentBytes
  • name: CONFIG_whisk_kafka_topics_health_retentionBytes
  • name: CONFIG_whisk_kafka_topics_health_retentionMs
  • name: CONFIG_whisk_kafka_topics_health_segmentBytes
  • name: CONFIG_whisk_kafka_topics_invoker_retentionBytes
  • name: CONFIG_whisk_kafka_topics_invoker_retentionMs
  • name: CONFIG_whisk_kafka_topics_invoker_segmentBytes
  • name: CONFIG_whisk_couchdb_username valueFrom: secretKeyRef: key: db_username name: openwhisk-db.auth
  • name: CONFIG_whisk_couchdb_password valueFrom: secretKeyRef: key: db_password name: openwhisk-db.auth
  • name: CONFIG_whisk_couchdb_port valueFrom: configMapKeyRef: key: db_port name: openwhisk-db.config
  • name: CONFIG_whisk_couchdb_protocol valueFrom: configMapKeyRef: key: db_protocol name: openwhisk-db.config
  • name: CONFIG_whisk_couchdb_host value: openwhisk-couchdb.openwhisk.svc.cluster.local
  • name: CONFIG_whisk_couchdb_provider valueFrom: configMapKeyRef: key: db_provider name: openwhisk-db.config
  • name: CONFIG_whisk_couchdb_databases_WhiskActivation valueFrom: configMapKeyRef: key: db_whisk_activations name: openwhisk-db.config
  • name: CONFIG_whisk_couchdb_databases_WhiskEntity valueFrom: configMapKeyRef: key: db_whisk_actions name: openwhisk-db.config
  • name: CONFIG_whisk_couchdb_databases_WhiskAuth valueFrom: configMapKeyRef: key: db_whisk_auths name: openwhisk-db.config
  • name: CONTROLLER_INSTANCES value: "1" image: openwhisk/controller:9bef49f imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /ping port: 8080 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: controller ports:
  • containerPort: 8080 name: controller protocol: TCP
  • containerPort: 2552 name: akka-remoting protocol: TCP
  • containerPort: 19999 name: akka-mgmt-http protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /ping port: 8080 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: {} securityContext: allowPrivilegeEscalation: false terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: openwhisk-core-token-x7snx readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true hostname: openwhisk-controller-0 initContainers:
• command:
  • sh
  • -c
  • result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 openwhisk-zookeeper-0.openwhisk-zookeeper.openwhisk.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi done; echo "Zookeeper is up; will wait for 10 seconds to give kafka time to initialize"; sleep 10; image: busybox imagePullPolicy: IfNotPresent name: wait-for-kafka resources: {} securityContext: allowPrivilegeEscalation: false terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: openwhisk-core-token-x7snx readOnly: true
• command:
  • sh
  • -c
  • 'while true; do echo ''checking CouchDB readiness''; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic xxx; result=$?; if [ $result -eq 0 ]; then echo ''Success: CouchDB is ready!''; break; fi; echo ''...not ready yet; sleeping 3 seconds before retry''; sleep 3; done;' env:
  • name: READINESS_URL value: http://openwhisk-couchdb.openwhisk.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker image: busybox imagePullPolicy: IfNotPresent name: wait-for-couchdb resources: {} securityContext: allowPrivilegeEscalation: false terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: openwhisk-core-token-x7snx readOnly: true nodeName: cdldvcldvkn0102 nodeSelector: beta.kubernetes.io/arch: amd64 priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: openwhisk-core serviceAccountName: openwhisk-core subdomain: openwhisk-controller terminationGracePeriodSeconds: 30 tolerations:
• effect: NoSchedule key: openwhisk-role operator: Equal value: core
• effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300
• effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes:
• name: openwhisk-core-token-x7snx secret: defaultMode: 420 secretName: openwhisk-core-token-x7snx status: conditions:
• lastProbeTime: null lastTransitionTime: "2020-09-08T03:55:35Z" status: "True" type: Initialized
• lastProbeTime: null lastTransitionTime: "2020-09-03T20:07:20Z" status: "True" type: Ready
• lastProbeTime: null lastTransitionTime: "2020-09-03T20:07:20Z" status: "True" type: ContainersReady
• lastProbeTime: null lastTransitionTime: "2020-09-03T20:06:54Z" status: "True" type: PodScheduled containerStatuses:
• containerID: docker://d6b434f632cb1ff8b2768f053f2829a2e98b5a41d2cfa9bafe458ebab0c08a93 image: openwhisk/controller:9bef49f imageID: docker-pullable://openwhisk/controller@sha256:38a81ec0f0561c00ae353b3dd72e5c9335cd63b569cc3ace2c010705b0155093 lastState: {} name: controller ready: true restartCount: 0 state: running: startedAt: "2020-09-03T20:07:08Z" hostIP: x.x.x.x initContainerStatuses:
• containerID: docker://ec41613ad52f6535a52f3ab1d78249b333a926d20a325a4026c4358743788010 image: busybox:latest imageID: docker-pullable://busybox@sha256:c3dbcbbf6261c620d133312aee9e858b45e1b686efbcead7b34d9aae58a37378 lastState: {} name: wait-for-kafka ready: true restartCount: 0 state: terminated: containerID: docker://ec41613ad52f6535a52f3ab1d78249b333a926d20a325a4026c4358743788010 exitCode: 0 finishedAt: "2020-09-08T03:55:35Z" reason: Completed startedAt: "2020-09-08T03:55:25Z"
• containerID: docker://23a7e568d1678d944054e701e88e9d1deb5e38edaf94d5a2f6e90fd195174237 image: busybox:latest imageID: docker-pullable://busybox@sha256:c3dbcbbf6261c620d133312aee9e858b45e1b686efbcead7b34d9aae58a37378 lastState: {} name: wait-for-couchdb ready: true restartCount: 0 state: terminated: containerID: docker://23a7e568d1678d944054e701e88e9d1deb5e38edaf94d5a2f6e90fd195174237 exitCode: 0 finishedAt: "2020-09-08T03:55:36Z" reason: Completed startedAt: "2020-09-08T03:55:36Z" phase: Running podIP: x.x.x.x qosClass: BestEffort startTime: "2020-09-03T20:06:54Z" `

[style95]

Those configurations are system-wide configurations and users can change action metadata within those limits. And your k8s configurations look fine on the surface. Are you still unable to update your action limits accordingly?

[rads18]

Just to clarify my initial post again the action allows me to set the new limits but fails with "Internal Whisk error", in env A. However, we did a fresh setup in another environment(B) recently where the new limits were setup during install and those work fine.

This means in existing env A we are missing some configuration file which does not allow the setup to work. We can reinstall the existing environment with new higher limits but i dont believe it to be a good solution, specially to reinstall for anytime the environment variables change.