Native-image Cassandra smoke test is flaky

[pvlugter]

The native-image smoke test for Cassandra is often failing and too unreliable. This test will be disabled until we can address this.

Example logs for failed test

``` #!/bin/bash -eo pipefail bin/run-java-eventsourced-shopping-cart-test.sh Cassandra --no-build === Running java-eventsourced-shopping-cart test === statefulstore.cloudstate.io/cassandra created statefulservice.cloudstate.io/shopping-cart-cassandra created Waiting for deployment to be created... NAME READY UP-TO-DATE AVAILABLE AGE shopping-cart-cassandra 0/1 1 0 1s Waiting for deployment to be ready... Waiting for deployment "shopping-cart-cassandra" rollout to finish: 0 of 1 updated replicas are available... deployment "shopping-cart-cassandra" successfully rolled out NAME READY UP-TO-DATE AVAILABLE AGE shopping-cart-cassandra 1/1 1 1 8s Scaling deployment... statefulservice.cloudstate.io/shopping-cart-cassandra scaled NAME READY UP-TO-DATE AVAILABLE AGE shopping-cart-cassandra 1/3 1 1 8s Waiting for deployment to be ready... Waiting for deployment "shopping-cart-cassandra" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment "shopping-cart-cassandra" rollout to finish: 2 of 3 updated replicas are available... deployment "shopping-cart-cassandra" successfully rolled out NAME READY UP-TO-DATE AVAILABLE AGE shopping-cart-cassandra 3/3 3 3 23s service/shopping-cart-cassandra-node-port exposed Testing eventsourced shopping cart test1 ... Expected '{"items":[]}' But got 'The server was not able to produce a timely response to your request. Please try again in a short while!' Failed: === Operator logs === 2021-02-02T00:03:21.517Z INFO setup using config {"file": "/etc/config/config.yaml"} 2021-02-02T00:03:22.172Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "127.0.0.1:8080"} 2021-02-02T00:03:22.172Z INFO controller-runtime.webhook registering webhook {"path": "/inject-v1-pod"} 2021-02-02T00:03:22.172Z INFO setup starting manager I0202 00:03:22.174217 1 leaderelection.go:242] attempting to acquire leader lease cloudstate-system/cloudstate-leader-election-helper... 2021-02-02T00:03:22.174Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"} 2021-02-02T00:03:22.175Z INFO controller-runtime.webhook.webhooks starting webhook server 2021-02-02T00:03:22.175Z INFO controller-runtime.certwatcher Updated current TLS certificate 2021-02-02T00:03:22.175Z INFO controller-runtime.webhook serving webhook server {"host": "", "port": 9443} 2021-02-02T00:03:22.217Z INFO controller-runtime.certwatcher Starting certificate watcher I0202 00:03:22.222430 1 leaderelection.go:252] successfully acquired lease cloudstate-system/cloudstate-leader-election-helper 2021-02-02T00:03:22.222Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"cloudstate-system","name":"cloudstate-leader-election-helper","uid":"2f295404-3bbd-4c05-bea8-39e7d27c1e81","apiVersion":"v1","resourceVersion":"715"}, "reason": "LeaderElection", "message": "cloudstate-controller-manager-648c764f7b-84nsh_0288eb0a-1691-41b7-8a73-8a9ca7b6da8b became leader"} 2021-02-02T00:03:22.222Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="} 2021-02-02T00:03:22.222Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore", "source": "kind source: /, Kind="} 2021-02-02T00:03:22.323Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="} 2021-02-02T00:03:22.323Z INFO controller Starting Controller {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore"} 2021-02-02T00:03:22.517Z INFO controller Starting workers {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore", "worker count": 1} 2021-02-02T00:03:22.517Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="} 2021-02-02T00:03:22.618Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="} 2021-02-02T00:03:22.718Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="} 2021-02-02T00:03:22.819Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="} 2021-02-02T00:03:22.919Z INFO controller Starting Controller {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice"} 2021-02-02T00:03:22.919Z INFO controller Starting workers {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "worker count": 1} 2021-02-02T00:03:27.020Z DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/inject-v1-pod", "UID": "e5137699-440f-41eb-a033-6c4a242cab6c", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}} 2021-02-02T00:03:27.026Z DEBUG controller-runtime.webhook.webhooks wrote response {"webhook": "/inject-v1-pod", "code": 200, "reason": "not a stateful service, not injecting", "UID": "e5137699-440f-41eb-a033-6c4a242cab6c", "allowed": true} 2021-02-02T00:04:45.218Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore", "name": "cassandra", "namespace": "default"} 2021-02-02T00:04:45.218Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore", "name": "cassandra", "namespace": "default"} 2021-02-02T00:04:45.543Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 0, "summary": "Unavailable"} 2021-02-02T00:04:45.548Z INFO controllers.StatefulService Created deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:45.548Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:04:45.549Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 0, "status.availableReplicas": 0} 2021-02-02T00:04:45.558Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 0, "summary": "UpdateInProgress"} 2021-02-02T00:04:45.560Z DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/inject-v1-pod", "UID": "574b962d-2e4d-4872-8cb0-16d964f7166c", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}} 2021-02-02T00:04:45.617Z DEBUG controller-runtime.webhook.webhooks wrote response {"webhook": "/inject-v1-pod", "code": 200, "reason": "", "UID": "574b962d-2e4d-4872-8cb0-16d964f7166c", "allowed": true} 2021-02-02T00:04:45.618Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:45.618Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:04:45.619Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 0, "status.availableReplicas": 0} 2021-02-02T00:04:45.619Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:45.621Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:45.621Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:04:45.646Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 0} 2021-02-02T00:04:45.652Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 0, "summary": "Unavailable"} 2021-02-02T00:04:45.717Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:45.717Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:04:45.718Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 0} 2021-02-02T00:04:45.718Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:45.719Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:45.719Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:04:53.465Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 1} 2021-02-02T00:04:53.517Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 1, "summary": "Ready"} 2021-02-02T00:04:53.519Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:53.519Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:04:53.520Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 1} 2021-02-02T00:04:53.520Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:53.522Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:53.522Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:04:53.676Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 1} 2021-02-02T00:04:53.676Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:53.679Z DEBUG controllers.StatefulService changed {"statefulservice": "default/shopping-cart-cassandra", "type": "deployment", "patchResult": "\nPatch: {\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"metadata\\\":{\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\"},\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"namespace\\\":\\\"default\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"cloudstate.io/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"StatefulService\\\",\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"uid\\\":\\\"28abd19d-86cc-4692-bf2a-015a54be1cd1\\\"}]},\\\"spec\\\":{\\\"replicas\\\":3,\\\"selector\\\":{\\\"matchLabels\\\":{\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"strategy\\\":{},\\\"template\\\":{\\\"metadata\\\":{\\\"annotations\\\":{\\\"cassandra.cloudstate.io/keyspace\\\":\\\"shoppingcart\\\",\\\"cloudstate.io/enabled\\\":\\\"true\\\",\\\"cloudstate.io/stateful-service-config\\\":\\\"ss-cfg-shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-store\\\":\\\"cassandra\\\"},\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"image\\\":\\\"cloudstateio/java-eventsourced-shopping-cart:latest\\\",\\\"imagePullPolicy\\\":\\\"Never\\\",\\\"name\\\":\\\"user-function\\\",\\\"resources\\\":{}}]}}},\\\"status\\\":{}}\"}},\"spec\":{\"replicas\":3}} \nCurrent: {\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"metadata\\\":{\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\"},\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"namespace\\\":\\\"default\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"cloudstate.io/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"StatefulService\\\",\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"uid\\\":\\\"28abd19d-86cc-4692-bf2a-015a54be1cd1\\\"}]},\\\"spec\\\":{\\\"replicas\\\":1,\\\"selector\\\":{\\\"matchLabels\\\":{\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"strategy\\\":{},\\\"template\\\":{\\\"metadata\\\":{\\\"annotations\\\":{\\\"cassandra.cloudstate.io/keyspace\\\":\\\"shoppingcart\\\",\\\"cloudstate.io/enabled\\\":\\\"true\\\",\\\"cloudstate.io/stateful-service-config\\\":\\\"ss-cfg-shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-store\\\":\\\"cassandra\\\"},\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"image\\\":\\\"cloudstateio/java-eventsourced-shopping-cart:latest\\\",\\\"imagePullPolicy\\\":\\\"Never\\\",\\\"name\\\":\\\"user-function\\\",\\\"resources\\\":{}}]}}},\\\"status\\\":{}}\",\"deployment.kubernetes.io/revision\":\"1\"},\"creationTimestamp\":\"2021-02-02T00:04:45Z\",\"generation\":1,\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\"},\"managedFields\":[{\"apiVersion\":\"apps/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:banzaicloud.com/last-applied\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/component\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"28abd19d-86cc-4692-bf2a-015a54be1cd1\\\"}\":{\".\":{},\"f:apiVersion\":{},\"f:blockOwnerDeletion\":{},\"f:controller\":{},\"f:kind\":{},\"f:name\":{},\"f:uid\":{}}}},\"f:spec\":{\"f:progressDeadlineSeconds\":{},\"f:replicas\":{},\"f:revisionHistoryLimit\":{},\"f:selector\":{\"f:matchLabels\":{\".\":{},\"f:cloudstate.io/stateful-service\":{}}},\"f:strategy\":{\"f:rollingUpdate\":{\".\":{},\"f:maxSurge\":{},\"f:maxUnavailable\":{}},\"f:type\":{}},\"f:template\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:cassandra.cloudstate.io/keyspace\":{},\"f:cloudstate.io/enabled\":{},\"f:cloudstate.io/stateful-service-config\":{},\"f:cloudstate.io/stateful-store\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/component\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{},\"f:cloudstate.io/stateful-service\":{}}},\"f:spec\":{\"f:containers\":{\"k:{\\\"name\\\":\\\"user-function\\\"}\":{\".\":{},\"f:image\":{},\"f:imagePullPolicy\":{},\"f:name\":{},\"f:resources\":{},\"f:terminationMessagePath\":{},\"f:terminationMessagePolicy\":{}}},\"f:dnsPolicy\":{},\"f:restartPolicy\":{},\"f:schedulerName\":{},\"f:securityContext\":{},\"f:terminationGracePeriodSeconds\":{}}}}},\"manager\":\"manager\",\"operation\":\"Update\",\"time\":\"2021-02-02T00:04:45Z\"},{\"apiVersion\":\"apps/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:deployment.kubernetes.io/revision\":{}}},\"f:status\":{\"f:availableReplicas\":{},\"f:conditions\":{\".\":{},\"k:{\\\"type\\\":\\\"Available\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"Progressing\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}},\"f:observedGeneration\":{},\"f:readyReplicas\":{},\"f:replicas\":{},\"f:updatedReplicas\":{}}},\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"time\":\"2021-02-02T00:04:53Z\"}],\"name\":\"shopping-cart-cassandra\",\"namespace\":\"default\",\"ownerReferences\":[{\"apiVersion\":\"cloudstate.io/v1alpha1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"StatefulService\",\"name\":\"shopping-cart-cassandra\",\"uid\":\"28abd19d-86cc-4692-bf2a-015a54be1cd1\"}],\"resourceVersion\":\"1014\",\"selfLink\":\"/apis/apps/v1/namespaces/default/deployments/shopping-cart-cassandra\",\"uid\":\"459e45de-a4a5-485a-ab9b-f296869812f3\"},\"spec\":{\"progressDeadlineSeconds\":600,\"replicas\":1,\"revisionHistoryLimit\":10,\"selector\":{\"matchLabels\":{\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"strategy\":{\"rollingUpdate\":{\"maxSurge\":\"25%\",\"maxUnavailable\":\"25%\"},\"type\":\"RollingUpdate\"},\"template\":{\"metadata\":{\"annotations\":{\"cassandra.cloudstate.io/keyspace\":\"shoppingcart\",\"cloudstate.io/enabled\":\"true\",\"cloudstate.io/stateful-service-config\":\"ss-cfg-shopping-cart-cassandra\",\"cloudstate.io/stateful-store\":\"cassandra\"},\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\",\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"spec\":{\"containers\":[{\"image\":\"cloudstateio/java-eventsourced-shopping-cart:latest\",\"imagePullPolicy\":\"Never\",\"name\":\"user-function\",\"resources\":{},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\"}],\"dnsPolicy\":\"ClusterFirst\",\"restartPolicy\":\"Always\",\"schedulerName\":\"default-scheduler\",\"securityContext\":{},\"terminationGracePeriodSeconds\":30}}},\"status\":{\"availableReplicas\":1,\"conditions\":[{\"lastTransitionTime\":\"2021-02-02T00:04:53Z\",\"lastUpdateTime\":\"2021-02-02T00:04:53Z\",\"message\":\"Deployment has minimum availability.\",\"reason\":\"MinimumReplicasAvailable\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2021-02-02T00:04:45Z\",\"lastUpdateTime\":\"2021-02-02T00:04:53Z\",\"message\":\"ReplicaSet \\\"shopping-cart-cassandra-65569f85cd\\\" has successfully progressed.\",\"reason\":\"NewReplicaSetAvailable\",\"status\":\"True\",\"type\":\"Progressing\"}],\"observedGeneration\":1,\"readyReplicas\":1,\"replicas\":1,\"updatedReplicas\":1}}\nModified: {\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"metadata\\\":{\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\"},\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"namespace\\\":\\\"default\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"cloudstate.io/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"StatefulService\\\",\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"uid\\\":\\\"28abd19d-86cc-4692-bf2a-015a54be1cd1\\\"}]},\\\"spec\\\":{\\\"replicas\\\":3,\\\"selector\\\":{\\\"matchLabels\\\":{\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"strategy\\\":{},\\\"template\\\":{\\\"metadata\\\":{\\\"annotations\\\":{\\\"cassandra.cloudstate.io/keyspace\\\":\\\"shoppingcart\\\",\\\"cloudstate.io/enabled\\\":\\\"true\\\",\\\"cloudstate.io/stateful-service-config\\\":\\\"ss-cfg-shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-store\\\":\\\"cassandra\\\"},\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"image\\\":\\\"cloudstateio/java-eventsourced-shopping-cart:latest\\\",\\\"imagePullPolicy\\\":\\\"Never\\\",\\\"name\\\":\\\"user-function\\\",\\\"resources\\\":{}}]}}},\\\"status\\\":{}}\"},\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\"},\"name\":\"shopping-cart-cassandra\",\"namespace\":\"default\",\"ownerReferences\":[{\"apiVersion\":\"cloudstate.io/v1alpha1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"StatefulService\",\"name\":\"shopping-cart-cassandra\",\"uid\":\"28abd19d-86cc-4692-bf2a-015a54be1cd1\"}]},\"spec\":{\"replicas\":3,\"selector\":{\"matchLabels\":{\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"strategy\":{},\"template\":{\"metadata\":{\"annotations\":{\"cassandra.cloudstate.io/keyspace\":\"shoppingcart\",\"cloudstate.io/enabled\":\"true\",\"cloudstate.io/stateful-service-config\":\"ss-cfg-shopping-cart-cassandra\",\"cloudstate.io/stateful-store\":\"cassandra\"},\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\",\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"spec\":{\"containers\":[{\"image\":\"cloudstateio/java-eventsourced-shopping-cart:latest\",\"imagePullPolicy\":\"Never\",\"name\":\"user-function\",\"resources\":{}}]}}},\"status\":{}}\nOriginal: {\"metadata\":{\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\"},\"name\":\"shopping-cart-cassandra\",\"namespace\":\"default\",\"ownerReferences\":[{\"apiVersion\":\"cloudstate.io/v1alpha1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"StatefulService\",\"name\":\"shopping-cart-cassandra\",\"uid\":\"28abd19d-86cc-4692-bf2a-015a54be1cd1\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"strategy\":{},\"template\":{\"metadata\":{\"annotations\":{\"cassandra.cloudstate.io/keyspace\":\"shoppingcart\",\"cloudstate.io/enabled\":\"true\",\"cloudstate.io/stateful-service-config\":\"ss-cfg-shopping-cart-cassandra\",\"cloudstate.io/stateful-store\":\"cassandra\"},\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\",\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"spec\":{\"containers\":[{\"image\":\"cloudstateio/java-eventsourced-shopping-cart:latest\",\"imagePullPolicy\":\"Never\",\"name\":\"user-function\",\"resources\":{}}]}}},\"status\":{}}\n"} 2021-02-02T00:04:53.684Z INFO controllers.StatefulService Updated deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:53.684Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:04:53.685Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 1, "status.availableReplicas": 1} 2021-02-02T00:04:53.702Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 1, "summary": "UpdateInProgress"} 2021-02-02T00:04:53.704Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:53.704Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:04:53.720Z DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/inject-v1-pod", "UID": "3b3c3740-345a-4a90-84fc-2e890ad2cda8", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}} 2021-02-02T00:04:53.721Z DEBUG controller-runtime.webhook.webhooks wrote response {"webhook": "/inject-v1-pod", "code": 200, "reason": "", "UID": "3b3c3740-345a-4a90-84fc-2e890ad2cda8", "allowed": true} 2021-02-02T00:04:53.723Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 1, "status.availableReplicas": 1} 2021-02-02T00:04:53.723Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:53.726Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:53.726Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:04:53.727Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 1, "status.availableReplicas": 1} 2021-02-02T00:04:53.729Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:53.732Z DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/inject-v1-pod", "UID": "54b8cc51-9755-4154-ab50-18ca941a8a7e", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}} 2021-02-02T00:04:53.817Z DEBUG controller-runtime.webhook.webhooks wrote response {"webhook": "/inject-v1-pod", "code": 200, "reason": "", "UID": "54b8cc51-9755-4154-ab50-18ca941a8a7e", "allowed": true} 2021-02-02T00:04:53.819Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:53.819Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:04:53.854Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 1} 2021-02-02T00:04:53.919Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 1, "summary": "PartiallyReady"} 2021-02-02T00:04:53.920Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:53.920Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:04:53.921Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 1} 2021-02-02T00:04:53.921Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:53.923Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:04:53.923Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:05:00.926Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 1} 2021-02-02T00:05:00.926Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:05:00.928Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:05:00.928Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:05:08.418Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 2} 2021-02-02T00:05:08.424Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 2, "summary": "PartiallyReady"} 2021-02-02T00:05:08.425Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:05:08.425Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:05:08.428Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 2} 2021-02-02T00:05:08.428Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:05:08.429Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:05:08.429Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:05:08.607Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 3} 2021-02-02T00:05:08.614Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 3, "summary": "Ready"} 2021-02-02T00:05:08.615Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:05:08.615Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} 2021-02-02T00:05:08.616Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 3} 2021-02-02T00:05:08.616Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:05:08.617Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"} 2021-02-02T00:05:08.617Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"} === Deployment description === Name: shopping-cart-cassandra Namespace: default CreationTimestamp: Tue, 02 Feb 2021 00:04:45 +0000 Labels: app.kubernetes.io/component=user-function app.kubernetes.io/managed-by=cloudstate-operator app.kubernetes.io/name=shopping-cart-cassandra Annotations: banzaicloud.com/last-applied: {"metadata":{"labels":{"app.kubernetes.io/component":"user-function","app.kubernetes.io/managed-by":"cloudstate-operator","app.kubernetes.... deployment.kubernetes.io/revision: 1 Selector: cloudstate.io/stateful-service=shopping-cart-cassandra Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app.kubernetes.io/component=user-function app.kubernetes.io/managed-by=cloudstate-operator app.kubernetes.io/name=shopping-cart-cassandra cloudstate.io/stateful-service=shopping-cart-cassandra Annotations: cassandra.cloudstate.io/keyspace: shoppingcart cloudstate.io/enabled: true cloudstate.io/stateful-service-config: ss-cfg-shopping-cart-cassandra cloudstate.io/stateful-store: cassandra Containers: user-function: Image: cloudstateio/java-eventsourced-shopping-cart:latest Port: Host Port: Environment: Mounts: Volumes: Conditions: Type Status Reason ---- ------ ------ Progressing True NewReplicaSetAvailable Available True MinimumReplicasAvailable OldReplicaSets: NewReplicaSet: shopping-cart-cassandra-65569f85cd (3/3 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 44s deployment-controller Scaled up replica set shopping-cart-cassandra-65569f85cd to 1 Normal ScalingReplicaSet 36s deployment-controller Scaled up replica set shopping-cart-cassandra-65569f85cd to 3 === Pods description === Name: cassandra-0 Namespace: default Priority: 0 Node: default-235cd9d9-df1d-4cc5-b459-28892cef4871/10.128.1.60 Start Time: Tue, 02 Feb 2021 00:03:27 +0000 Labels: app=cassandra chart=cassandra-5.6.1 controller-revision-hash=cassandra-67cf96f4b8 heritage=Helm release=cassandra statefulset.kubernetes.io/pod-name=cassandra-0 Annotations: Status: Running IP: 172.17.0.7 IPs: IP: 172.17.0.7 Controlled By: StatefulSet/cassandra Containers: cassandra: Container ID: docker://ef7252fa690d037a6a83ee7c8334e33256c2ad57a742fb1c4eda6d9e5d39b58f Image: docker.io/bitnami/cassandra:3.11.7-debian-10-r4 Image ID: docker-pullable://bitnami/cassandra@sha256:c864afbe56596553c72a3babf9c72158e89241a7e786eddbc2fa2551b56403ef Ports: 7000/TCP, 7001/TCP, 7199/TCP, 9042/TCP, 9160/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP Command: bash -ec if [[ $HOSTNAME =~ (.*)-0$ ]]; then echo "Setting node as password seeder" export CASSANDRA_PASSWORD_SEEDER=yes else # Only node 0 will execute the startup initdb scripts export CASSANDRA_IGNORE_INITDB_SCRIPTS=1 fi /entrypoint.sh /run.sh State: Running Started: Tue, 02 Feb 2021 00:03:40 +0000 Ready: True Restart Count: 0 Liveness: exec [/bin/sh -c nodetool status] delay=60s timeout=5s period=30s #success=1 #failure=5 Readiness: exec [/bin/sh -c nodetool status | grep -E "^UN\s+${POD_IP}"] delay=60s timeout=5s period=10s #success=1 #failure=5 Environment: BITNAMI_DEBUG: false CASSANDRA_CLUSTER_NAME: cassandra CASSANDRA_SEEDS: cassandra-0.cassandra-headless.default.svc.cluster.local CASSANDRA_PASSWORD: Optional: false POD_IP: (v1:status.podIP) CASSANDRA_USER: cassandra CASSANDRA_NUM_TOKENS: 256 CASSANDRA_DATACENTER: dc1 CASSANDRA_ENDPOINT_SNITCH: SimpleSnitch CASSANDRA_ENDPOINT_SNITCH: SimpleSnitch CASSANDRA_RACK: rack1 CASSANDRA_ENABLE_RPC: true Mounts: /bitnami/cassandra from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: data: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: default-token-wd7lz: Type: Secret (a volume populated by a Secret) SecretName: default-token-wd7lz Optional: false QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m2s default-scheduler Successfully assigned default/cassandra-0 to default-235cd9d9-df1d-4cc5-b459-28892cef4871 Normal Pulling 2m2s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Pulling image "docker.io/bitnami/cassandra:3.11.7-debian-10-r4" Normal Pulled 109s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Successfully pulled image "docker.io/bitnami/cassandra:3.11.7-debian-10-r4" Normal Created 109s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container cassandra Normal Started 109s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container cassandra Name: shopping-cart-cassandra-65569f85cd-dxpbr Namespace: default Priority: 0 Node: default-235cd9d9-df1d-4cc5-b459-28892cef4871/10.128.1.60 Start Time: Tue, 02 Feb 2021 00:04:53 +0000 Labels: app.kubernetes.io/component=user-function app.kubernetes.io/managed-by=cloudstate-operator app.kubernetes.io/name=shopping-cart-cassandra cloudstate.io/stateful-service=shopping-cart-cassandra pod-template-hash=65569f85cd Annotations: cassandra.cloudstate.io/keyspace: shoppingcart cloudstate.io/enabled: true cloudstate.io/stateful-service-config: ss-cfg-shopping-cart-cassandra cloudstate.io/stateful-store: cassandra traffic.sidecar.istio.io/excludeInboundPorts: 2552,8558 traffic.sidecar.istio.io/excludeOutboundPorts: 2552,8558 Status: Running IP: 172.17.0.10 IPs: IP: 172.17.0.10 Controlled By: ReplicaSet/shopping-cart-cassandra-65569f85cd Containers: user-function: Container ID: docker://062c42b378755454c4ca7be16a1f818bf62cfca5203bce8330133b9ac75dc40a Image: cloudstateio/java-eventsourced-shopping-cart:latest Image ID: docker://sha256:3df7437cc0f3bce3c4a5c9ff7293a884ce2e61a29cadbee171a849f204c5c35f Port: 8080/TCP Host Port: 0/TCP State: Running Started: Tue, 02 Feb 2021 00:04:54 +0000 Ready: True Restart Count: 0 Environment: PORT: 8080 Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro) cloudstate-sidecar: Container ID: docker://4b0f0cc559a8517acd98ab2309470d97987af5a41d68a942e99fa910ddc1b213 Image: cloudstateio/cloudstate-proxy-native-cassandra:latest Image ID: docker://sha256:0ba59b5fc83c69f703d7703cb92630c031944712d61d91efd14ec6056f249559 Ports: 8013/TCP, 9099/TCP Host Ports: 0/TCP, 0/TCP State: Running Started: Tue, 02 Feb 2021 00:04:54 +0000 Ready: True Restart Count: 0 Limits: memory: 512Mi Requests: cpu: 400m memory: 512Mi Liveness: http-get http://:8558/alive delay=0s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:8558/ready delay=0s timeout=1s period=10s #success=1 #failure=5 Environment: USER_FUNCTION_PORT: 8080 METRICS_PORT: 9099 REMOTING_PORT: 2552 MANAGEMENT_PORT: 8558 HTTP_PORT: 8013 SELECTOR_LABEL: cloudstate.io/stateful-service SELECTOR_LABEL_VALUE: shopping-cart-cassandra REQUIRED_CONTACT_POINT_NR: 1 CASSANDRA_CONTACT_POINTS: cassandra.default.svc.cluster.local CASSANDRA_KEYSPACE: shoppingcart CASSANDRA_USERNAME: Optional: false CASSANDRA_PASSWORD: Optional: false Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-wd7lz: Type: Secret (a volume populated by a Secret) SecretName: default-token-wd7lz Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 36s default-scheduler Successfully assigned default/shopping-cart-cassandra-65569f85cd-dxpbr to default-235cd9d9-df1d-4cc5-b459-28892cef4871 Normal Pulled 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Container image "cloudstateio/java-eventsourced-shopping-cart:latest" already present on machine Normal Created 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container user-function Normal Started 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container user-function Normal Pulled 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Container image "cloudstateio/cloudstate-proxy-native-cassandra:latest" already present on machine Normal Created 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container cloudstate-sidecar Normal Started 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container cloudstate-sidecar Warning Unhealthy 31s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Readiness probe failed: HTTP probe failed with statuscode: 500 Name: shopping-cart-cassandra-65569f85cd-g4vq5 Namespace: default Priority: 0 Node: default-235cd9d9-df1d-4cc5-b459-28892cef4871/10.128.1.60 Start Time: Tue, 02 Feb 2021 00:04:45 +0000 Labels: app.kubernetes.io/component=user-function app.kubernetes.io/managed-by=cloudstate-operator app.kubernetes.io/name=shopping-cart-cassandra cloudstate.io/stateful-service=shopping-cart-cassandra pod-template-hash=65569f85cd Annotations: cassandra.cloudstate.io/keyspace: shoppingcart cloudstate.io/enabled: true cloudstate.io/stateful-service-config: ss-cfg-shopping-cart-cassandra cloudstate.io/stateful-store: cassandra traffic.sidecar.istio.io/excludeInboundPorts: 2552,8558 traffic.sidecar.istio.io/excludeOutboundPorts: 2552,8558 Status: Running IP: 172.17.0.8 IPs: IP: 172.17.0.8 Controlled By: ReplicaSet/shopping-cart-cassandra-65569f85cd Containers: user-function: Container ID: docker://b238c07e01234ef592fd85ca9f08d3902bc447c222911429e09a8044991461cc Image: cloudstateio/java-eventsourced-shopping-cart:latest Image ID: docker://sha256:3df7437cc0f3bce3c4a5c9ff7293a884ce2e61a29cadbee171a849f204c5c35f Port: 8080/TCP Host Port: 0/TCP State: Running Started: Tue, 02 Feb 2021 00:04:46 +0000 Ready: True Restart Count: 0 Environment: PORT: 8080 Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro) cloudstate-sidecar: Container ID: docker://a97aca57bb76b2fe1e9913d5879dddab0f108e8fd0cee85933780d108bcffd7b Image: cloudstateio/cloudstate-proxy-native-cassandra:latest Image ID: docker://sha256:0ba59b5fc83c69f703d7703cb92630c031944712d61d91efd14ec6056f249559 Ports: 8013/TCP, 9099/TCP Host Ports: 0/TCP, 0/TCP State: Running Started: Tue, 02 Feb 2021 00:04:46 +0000 Ready: True Restart Count: 0 Limits: memory: 512Mi Requests: cpu: 400m memory: 512Mi Liveness: http-get http://:8558/alive delay=0s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:8558/ready delay=0s timeout=1s period=10s #success=1 #failure=5 Environment: USER_FUNCTION_PORT: 8080 METRICS_PORT: 9099 REMOTING_PORT: 2552 MANAGEMENT_PORT: 8558 HTTP_PORT: 8013 SELECTOR_LABEL: cloudstate.io/stateful-service SELECTOR_LABEL_VALUE: shopping-cart-cassandra REQUIRED_CONTACT_POINT_NR: 1 CASSANDRA_CONTACT_POINTS: cassandra.default.svc.cluster.local CASSANDRA_KEYSPACE: shoppingcart CASSANDRA_USERNAME: Optional: false CASSANDRA_PASSWORD: Optional: false Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-wd7lz: Type: Secret (a volume populated by a Secret) SecretName: default-token-wd7lz Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 44s default-scheduler Successfully assigned default/shopping-cart-cassandra-65569f85cd-g4vq5 to default-235cd9d9-df1d-4cc5-b459-28892cef4871 Normal Pulled 43s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Container image "cloudstateio/java-eventsourced-shopping-cart:latest" already present on machine Normal Created 43s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container user-function Normal Started 43s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container user-function Normal Pulled 43s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Container image "cloudstateio/cloudstate-proxy-native-cassandra:latest" already present on machine Normal Created 43s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container cloudstate-sidecar Normal Started 43s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container cloudstate-sidecar Name: shopping-cart-cassandra-65569f85cd-q6jd5 Namespace: default Priority: 0 Node: default-235cd9d9-df1d-4cc5-b459-28892cef4871/10.128.1.60 Start Time: Tue, 02 Feb 2021 00:04:53 +0000 Labels: app.kubernetes.io/component=user-function app.kubernetes.io/managed-by=cloudstate-operator app.kubernetes.io/name=shopping-cart-cassandra cloudstate.io/stateful-service=shopping-cart-cassandra pod-template-hash=65569f85cd Annotations: cassandra.cloudstate.io/keyspace: shoppingcart cloudstate.io/enabled: true cloudstate.io/stateful-service-config: ss-cfg-shopping-cart-cassandra cloudstate.io/stateful-store: cassandra traffic.sidecar.istio.io/excludeInboundPorts: 2552,8558 traffic.sidecar.istio.io/excludeOutboundPorts: 2552,8558 Status: Running IP: 172.17.0.9 IPs: IP: 172.17.0.9 Controlled By: ReplicaSet/shopping-cart-cassandra-65569f85cd Containers: user-function: Container ID: docker://2f1dc02b9a6254404e8bcce1489d2c4c7b02ea01f3fb0c7729922cdfbf68ce7e Image: cloudstateio/java-eventsourced-shopping-cart:latest Image ID: docker://sha256:3df7437cc0f3bce3c4a5c9ff7293a884ce2e61a29cadbee171a849f204c5c35f Port: 8080/TCP Host Port: 0/TCP State: Running Started: Tue, 02 Feb 2021 00:04:54 +0000 Ready: True Restart Count: 0 Environment: PORT: 8080 Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro) cloudstate-sidecar: Container ID: docker://b30cac0bd4cd0027d7773b3c401900438232779e633ae2f3cac54d0f3cd040c5 Image: cloudstateio/cloudstate-proxy-native-cassandra:latest Image ID: docker://sha256:0ba59b5fc83c69f703d7703cb92630c031944712d61d91efd14ec6056f249559 Ports: 8013/TCP, 9099/TCP Host Ports: 0/TCP, 0/TCP State: Running Started: Tue, 02 Feb 2021 00:04:54 +0000 Ready: True Restart Count: 0 Limits: memory: 512Mi Requests: cpu: 400m memory: 512Mi Liveness: http-get http://:8558/alive delay=0s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:8558/ready delay=0s timeout=1s period=10s #success=1 #failure=5 Environment: USER_FUNCTION_PORT: 8080 METRICS_PORT: 9099 REMOTING_PORT: 2552 MANAGEMENT_PORT: 8558 HTTP_PORT: 8013 SELECTOR_LABEL: cloudstate.io/stateful-service SELECTOR_LABEL_VALUE: shopping-cart-cassandra REQUIRED_CONTACT_POINT_NR: 1 CASSANDRA_CONTACT_POINTS: cassandra.default.svc.cluster.local CASSANDRA_KEYSPACE: shoppingcart CASSANDRA_USERNAME: Optional: false CASSANDRA_PASSWORD: Optional: false Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-wd7lz: Type: Secret (a volume populated by a Secret) SecretName: default-token-wd7lz Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 36s default-scheduler Successfully assigned default/shopping-cart-cassandra-65569f85cd-q6jd5 to default-235cd9d9-df1d-4cc5-b459-28892cef4871 Normal Pulled 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Container image "cloudstateio/java-eventsourced-shopping-cart:latest" already present on machine Normal Created 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container user-function Normal Started 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container user-function Normal Pulled 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Container image "cloudstateio/cloudstate-proxy-native-cassandra:latest" already present on machine Normal Created 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container cloudstate-sidecar Normal Started 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container cloudstate-sidecar Warning Unhealthy 31s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Readiness probe failed: HTTP probe failed with statuscode: 500 === Proxy logs === 2021-02-02 00:04:54.921 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2021-02-02 00:04:54.944 WARN akka.actor.ActorSystemImpl - Failed to load JFR flight recorder, falling back to noop. Exception: java.lang.ClassNotFoundException: akka.remote.artery.jfr.JFRRemotingFlightRecorder 2021-02-02 00:04:54.949 INFO akka.remote.artery.tcp.ArteryTcpTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cloudstate-proxy@172.17.0.10:2552] with UID [5053097932859970139] 2021-02-02 00:04:54.950 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.10:2552] - Starting up, Akka version [2.6.9] ... 2021-02-02 00:04:54.951 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.10:2552] - Started up successfully 2021-02-02 00:04:54.957 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.10:2552] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 2021-02-02 00:04:54.958 INFO akka.cluster.sbr.SplitBrainResolver - SBR started. Config: stableAfter: 20000 ms, strategy: KeepMajority, selfUniqueAddress: UniqueAddress(akka://cloudstate-proxy@172.17.0.10:2552,5053097932859970139), selfDc: default 2021-02-02 00:04:54.968 INFO akka.actor.ActorSystemImpl - Starting Cloudstate Proxy version [0.6.0-5-f0446998] running on [Substrate VM GraalVM 20.3.1 Java 11] 2021-02-02 00:04:54.968 INFO akka.actor.ActorSystemImpl - Registering SIGTERM handler... 2021-02-02 00:04:54.968 INFO akka.management.internal.HealthChecksImpl - Loading readiness checks [(cloudstate,io.cloudstate.proxy.HealthCheckReady)] 2021-02-02 00:04:54.968 INFO akka.management.internal.HealthChecksImpl - Loading liveness checks [(cloudstate,io.cloudstate.proxy.HealthCheckLive)] 2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Binding Akka Management (HTTP) endpoint to: 172.17.0.10:8558 2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for ClusterBootstrap 2021-02-02 00:04:54.969 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Using self contact point address: http://172.17.0.10:8558 2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for HealthCheckRoutes 2021-02-02 00:04:54.975 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Initiating bootstrap procedure using kubernetes-api method... 2021-02-02 00:04:54.975 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Bootstrap using `akka.discovery` method: kubernetes-api 2021-02-02 00:04:54.975 INFO akka.management.scaladsl.AkkaManagement - Bound Akka Management (HTTP) endpoint to: 172.17.0.10:8558 2021-02-02 00:04:54.978 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Locating service members. Using discovery [akka.discovery.kubernetes.KubernetesApiServiceDiscovery], join decider [akka.management.cluster.bootstrap.LowestAddressJoinDecider], scheme [http] 2021-02-02 00:04:54.978 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))] 2021-02-02 00:04:54.978 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None] 2021-02-02 00:04:55.025 INFO akka.actor.ActorSystemImpl - Prometheus exporter started on /0.0.0.0:9099 2021-02-02 00:04:55.052 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at io.grpc.Status.asRuntimeException(Status.java:533) at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426) at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:834) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192) Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080 Caused by: java.net.ConnectException: Connection refused at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0) at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:834) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192) 2021-02-02 00:04:55.073 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8))], filtered to [172-17-0-8.default.pod.cluster.local:0] 2021-02-02 00:04:55.089 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Contact point [akka://cloudstate-proxy@172.17.0.8:2552] returned [1] seed-nodes [akka://cloudstate-proxy@172.17.0.8:2552] 2021-02-02 00:04:55.089 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Joining [akka://cloudstate-proxy@172.17.0.10:2552] to existing cluster [akka://cloudstate-proxy@172.17.0.8:2552] 2021-02-02 00:04:55.089 INFO akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [akka.management.cluster.bootstrap.internal.BootstrapCoordinator$Protocol$BootstrappingCompleted$] from Actor[akka://cloudstate-proxy/system/bootstrapCoordinator#1156786877] to Actor[akka://cloudstate-proxy/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 2021-02-02 00:04:55.249 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.10:2552] - Received InitJoinAck message from [Actor[akka://cloudstate-proxy@172.17.0.8:2552/system/cluster/core/daemon#-754755589]] to [akka://cloudstate-proxy@172.17.0.10:2552] 2021-02-02 00:04:55.375 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.10:2552] - Welcome from [akka://cloudstate-proxy@172.17.0.8:2552] 2021-02-02 00:04:55.978 INFO akka.cluster.sbr.SplitBrainResolver - This node is now the leader responsible for taking SBR decisions among the reachable nodes (more leaders may exist). 2021-02-02 00:04:56.257 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at io.grpc.Status.asRuntimeException(Status.java:533) at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426) at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:834) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192) Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080 Caused by: java.net.ConnectException: Connection refused at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0) at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:834) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192) 2021-02-02 00:04:56.281 INFO akka.actor.EmptyLocalActorRef - Message [akka.cluster.ddata.Replicator$Internal$Status] from Actor[akka://cloudstate-proxy@172.17.0.8:2552/system/sharding/replicator#-1039544820] to Actor[akka://cloudstate-proxy/system/sharding/replicator] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/system/sharding/replicator] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 2021-02-02 00:04:56.995 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.10:2552] - is the new leader among reachable nodes (more leaders may exist) 2021-02-02 00:04:58.300 INFO akka.actor.EmptyLocalActorRef - Message [akka.cluster.ddata.Replicator$Internal$Status] from Actor[akka://cloudstate-proxy@172.17.0.8:2552/system/sharding/replicator#-1039544820] to Actor[akka://cloudstate-proxy/system/sharding/replicator] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/system/sharding/replicator] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 2021-02-02 00:04:58.391 DEBUG io.cloudstate.proxy.HealthCheckReady - Error performing server manager readiness check akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://cloudstate-proxy/), Path(/user/server-manager-supervisor/server-manager)] at akka.actor.ActorSelection.$anonfun$resolveOne$1(ActorSelection.scala:74) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:447) at scala.concurrent.ExecutionContext$parasitic$.execute(ExecutionContext.scala:164) at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:392) at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:302) at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:276) at scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:180) at akka.actor.ActorSelection.resolveOne(ActorSelection.scala:72) at io.cloudstate.proxy.HealthCheckReady.check(CloudStateProxyMain.scala:48) at io.cloudstate.proxy.HealthCheckReady.apply(CloudStateProxyMain.scala:61) at io.cloudstate.proxy.HealthCheckReady.apply(CloudStateProxyMain.scala:38) at scala.util.Try$.apply(Try.scala:210) at akka.management.internal.HealthChecksImpl.runCheck(HealthChecksImpl.scala:177) at akka.management.internal.HealthChecksImpl.$anonfun$check$2(HealthChecksImpl.scala:195) at scala.collection.immutable.List.map(List.scala:246) at scala.collection.immutable.List.map(List.scala:79) at akka.management.internal.HealthChecksImpl.check(HealthChecksImpl.scala:185) at akka.management.internal.HealthChecksImpl.readyResult(HealthChecksImpl.scala:149) at akka.management.HealthCheckRoutes.$anonfun$routes$3(HealthCheckRoutes.scala:46) at akka.http.scaladsl.server.directives.FutureDirectives.$anonfun$onComplete$2(FutureDirectives.scala:37) at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:68) at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161) at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:45) at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161) at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47) at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41) at akka.http.scaladsl.util.FastFuture$.transformWith$extension(FastFuture.scala:45) at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26) at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44) at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44) at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47) at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41) at akka.http.scaladsl.util.FastFuture$.transformWith$extension(FastFuture.scala:45) at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26) at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44) at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:45) at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161) at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:74) at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161) at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32) at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161) at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86) at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1400) at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:541) at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:423) at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:625) at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:502) at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:600) at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:769) at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:784) at akka.actor.Actor.aroundReceive(Actor.scala:537) at akka.actor.Actor.aroundReceive$(Actor.scala:535) at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:691) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577) at akka.actor.ActorCell.invoke(ActorCell.scala:547) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) at akka.dispatch.Mailbox.run(Mailbox.scala:231) at akka.dispatch.Mailbox.exec(Mailbox.scala:243) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192) 2021-02-02 00:04:58.392 INFO akka.management.internal.HealthChecksImpl - Check [io.cloudstate.proxy.HealthCheckReady] not ok 2021-02-02 00:04:58.676 INFO akka.actor.EmptyLocalActorRef - Message [akka.cluster.ddata.Replicator$Internal$Write] from Actor[akka://cloudstate-proxy@172.17.0.8:2552/system/sharding/replicator/$c#-1166386490] to Actor[akka://cloudstate-proxy/system/sharding/replicator] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/system/sharding/replicator] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 2021-02-02 00:04:58.981 INFO io.cloudstate.proxy.EntityDiscoveryManager - Received EntitySpec from user function with info: ServiceInfo(,,OpenJDK Runtime Environment 11.0.10+9,cloudstate-java-support,0.6.0-4-f0446998,0,2,UnknownFieldSet(Map())) 2021-02-02 00:04:58.981 DEBUG io.cloudstate.proxy.eventsourced.EventSourcedSupportFactory - Starting EventSourcedEntity for eventsourced-shopping-cart 2021-02-02 00:04:58.982 INFO EventingManager - Eventing support turned off in configuration 2021-02-02 00:04:58.982 INFO akka.cluster.singleton.ClusterSingletonManager - ClusterSingletonManager state change [Start -> Younger] 2021-02-02 00:04:58.982 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/add),*,,Vector(),UnknownFieldSet(Map()))] 2021-02-02 00:04:58.983 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/{product_id}/remove),,,Vector(),UnknownFieldSet(Map()))] 2021-02-02 00:04:58.983 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Get(/carts/{user_id}),,,Vector(HttpRule(,Get(/carts/{user_id}/items),,items,Vector(),UnknownFieldSet(Map()))),UnknownFieldSet(Map()))] 2021-02-02 00:04:58.984 DEBUG io.cloudstate.proxy.EntityDiscoveryManager - Starting gRPC proxy 2021-02-02 00:04:58.987 INFO io.cloudstate.proxy.EntityDiscoveryManager - CloudState proxy online at /0.0.0.0:8013 2021-02-02 00:04:46.680 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2021-02-02 00:04:46.691 WARN akka.actor.ActorSystemImpl - Failed to load JFR flight recorder, falling back to noop. Exception: java.lang.ClassNotFoundException: akka.remote.artery.jfr.JFRRemotingFlightRecorder 2021-02-02 00:04:46.697 INFO akka.remote.artery.tcp.ArteryTcpTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cloudstate-proxy@172.17.0.8:2552] with UID [1154167314093528637] 2021-02-02 00:04:46.697 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Starting up, Akka version [2.6.9] ... 2021-02-02 00:04:46.698 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Started up successfully 2021-02-02 00:04:46.699 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 2021-02-02 00:04:46.699 INFO akka.cluster.sbr.SplitBrainResolver - SBR started. Config: stableAfter: 20000 ms, strategy: KeepMajority, selfUniqueAddress: UniqueAddress(akka://cloudstate-proxy@172.17.0.8:2552,1154167314093528637), selfDc: default 2021-02-02 00:04:46.716 INFO akka.actor.ActorSystemImpl - Starting Cloudstate Proxy version [0.6.0-5-f0446998] running on [Substrate VM GraalVM 20.3.1 Java 11] 2021-02-02 00:04:46.716 INFO akka.actor.ActorSystemImpl - Registering SIGTERM handler... 2021-02-02 00:04:46.716 INFO akka.management.internal.HealthChecksImpl - Loading readiness checks [(cloudstate,io.cloudstate.proxy.HealthCheckReady)] 2021-02-02 00:04:46.716 INFO akka.management.internal.HealthChecksImpl - Loading liveness checks [(cloudstate,io.cloudstate.proxy.HealthCheckLive)] 2021-02-02 00:04:46.719 INFO akka.management.scaladsl.AkkaManagement - Binding Akka Management (HTTP) endpoint to: 172.17.0.8:8558 2021-02-02 00:04:46.719 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for ClusterBootstrap 2021-02-02 00:04:46.719 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Using self contact point address: http://172.17.0.8:8558 2021-02-02 00:04:46.719 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for HealthCheckRoutes 2021-02-02 00:04:46.724 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Initiating bootstrap procedure using kubernetes-api method... 2021-02-02 00:04:46.724 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Bootstrap using `akka.discovery` method: kubernetes-api 2021-02-02 00:04:46.724 INFO akka.management.scaladsl.AkkaManagement - Bound Akka Management (HTTP) endpoint to: 172.17.0.8:8558 2021-02-02 00:04:46.729 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Locating service members. Using discovery [akka.discovery.kubernetes.KubernetesApiServiceDiscovery], join decider [akka.management.cluster.bootstrap.LowestAddressJoinDecider], scheme [http] 2021-02-02 00:04:46.729 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))] 2021-02-02 00:04:46.729 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None] 2021-02-02 00:04:46.731 INFO akka.actor.ActorSystemImpl - Prometheus exporter started on /0.0.0.0:9099 2021-02-02 00:04:46.765 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at io.grpc.Status.asRuntimeException(Status.java:533) at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426) at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:834) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192) Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080 Caused by: java.net.ConnectException: Connection refused at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0) at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:834) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192) 2021-02-02 00:04:46.786 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - No targets found from pod list. Is the correct port name configured? Current configuration: [None]. Ports on pods: [Set(ContainerPort(Some(user-port),8080), ContainerPort(Some(grpc-http-proxy),8013), ContainerPort(Some(cs-metrics),9099))] 2021-02-02 00:04:46.786 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [], filtered to [] 2021-02-02 00:04:47.740 INFO akka.management.cluster.bootstrap.LowestAddressJoinDecider - Discovered [0] contact points, confirmed [0], which is less than the required [1], retrying 2021-02-02 00:04:47.811 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at io.grpc.Status.asRuntimeException(Status.java:533) at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426) at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:834) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192) Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080 Caused by: java.net.ConnectException: Connection refused at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0) at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:834) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192) 2021-02-02 00:04:47.930 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))] 2021-02-02 00:04:47.930 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None] 2021-02-02 00:04:47.934 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8))], filtered to [172-17-0-8.default.pod.cluster.local:0] 2021-02-02 00:04:47.955 INFO akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes - Bootstrap request from 172.17.0.8:48894: Contact Point returning 0 seed-nodes [] 2021-02-02 00:04:47.956 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Contact point [akka://cloudstate-proxy@172.17.0.8:2552] returned [0] seed-nodes [] 2021-02-02 00:04:48.980 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))] 2021-02-02 00:04:48.981 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None] 2021-02-02 00:04:48.992 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8))], filtered to [172-17-0-8.default.pod.cluster.local:0] 2021-02-02 00:04:49.080 INFO akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes - Bootstrap request from 172.17.0.8:48894: Contact Point returning 0 seed-nodes [] 2021-02-02 00:04:49.080 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Contact point [akka://cloudstate-proxy@172.17.0.8:2552] returned [0] seed-nodes [] 2021-02-02 00:04:49.739 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Initiating new cluster, self-joining [akka://cloudstate-proxy@172.17.0.8:2552]. Other nodes are expected to locate this cluster via continued contact-point probing. 2021-02-02 00:04:49.740 INFO akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [akka.management.cluster.bootstrap.internal.BootstrapCoordinator$Protocol$BootstrappingCompleted$] from Actor[akka://cloudstate-proxy/system/bootstrapCoordinator#1480424403] to Actor[akka://cloudstate-proxy/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 2021-02-02 00:04:49.740 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Node [akka://cloudstate-proxy@172.17.0.8:2552] is JOINING itself (with roles [dc-default]) and forming new cluster 2021-02-02 00:04:49.740 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - is the new leader among reachable nodes (more leaders may exist) 2021-02-02 00:04:49.740 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Leader is moving node [akka://cloudstate-proxy@172.17.0.8:2552] to [Up] 2021-02-02 00:04:49.740 INFO akka.cluster.sbr.SplitBrainResolver - This node is now the leader responsible for taking SBR decisions among the reachable nodes (more leaders may exist). 2021-02-02 00:04:50.235 INFO io.cloudstate.proxy.EntityDiscoveryManager - Received EntitySpec from user function with info: ServiceInfo(,,OpenJDK Runtime Environment 11.0.10+9,cloudstate-java-support,0.6.0-4-f0446998,0,2,UnknownFieldSet(Map())) 2021-02-02 00:04:50.236 DEBUG io.cloudstate.proxy.eventsourced.EventSourcedSupportFactory - Starting EventSourcedEntity for eventsourced-shopping-cart 2021-02-02 00:04:50.237 INFO EventingManager - Eventing support turned off in configuration 2021-02-02 00:04:50.238 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/add),*,,Vector(),UnknownFieldSet(Map()))] 2021-02-02 00:04:50.238 INFO akka.cluster.singleton.ClusterSingletonManager - Singleton manager starting singleton actor [akka://cloudstate-proxy/system/sharding/eventsourced-shopping-cartCoordinator/singleton] 2021-02-02 00:04:50.238 INFO akka.cluster.singleton.ClusterSingletonManager - ClusterSingletonManager state change [Start -> Oldest] 2021-02-02 00:04:50.238 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/{product_id}/remove),,,Vector(),UnknownFieldSet(Map()))] 2021-02-02 00:04:50.238 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Get(/carts/{user_id}),,,Vector(HttpRule(,Get(/carts/{user_id}/items),,items,Vector(),UnknownFieldSet(Map()))),UnknownFieldSet(Map()))] 2021-02-02 00:04:50.238 INFO akka.cluster.sharding.DDataShardCoordinator - ShardCoordinator was moved to the active state State(Map()) 2021-02-02 00:04:50.239 DEBUG io.cloudstate.proxy.EntityDiscoveryManager - Starting gRPC proxy 2021-02-02 00:04:50.241 INFO io.cloudstate.proxy.EntityDiscoveryManager - CloudState proxy online at /0.0.0.0:8013 2021-02-02 00:04:55.086 INFO akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes - Bootstrap request from 172.17.0.9:50312: Contact Point returning 1 seed-nodes [akka://cloudstate-proxy@172.17.0.8:2552] 2021-02-02 00:04:55.088 INFO akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes - Bootstrap request from 172.17.0.10:58948: Contact Point returning 1 seed-nodes [akka://cloudstate-proxy@172.17.0.8:2552] 2021-02-02 00:04:55.125 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Received InitJoin message from [Actor[akka://cloudstate-proxy@172.17.0.10:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#188025117]] to [akka://cloudstate-proxy@172.17.0.8:2552] 2021-02-02 00:04:55.125 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Sending InitJoinAck message from node [akka://cloudstate-proxy@172.17.0.8:2552] to [Actor[akka://cloudstate-proxy@172.17.0.10:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#188025117]] (version [2.6.9]) 2021-02-02 00:04:55.249 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Received InitJoin message from [Actor[akka://cloudstate-proxy@172.17.0.9:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#933739104]] to [akka://cloudstate-proxy@172.17.0.8:2552] 2021-02-02 00:04:55.249 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Sending InitJoinAck message from node [akka://cloudstate-proxy@172.17.0.8:2552] to [Actor[akka://cloudstate-proxy@172.17.0.9:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#933739104]] (version [2.6.9]) 2021-02-02 00:04:55.374 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Node [akka://cloudstate-proxy@172.17.0.10:2552] is JOINING, roles [dc-default] 2021-02-02 00:04:55.379 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Node [akka://cloudstate-proxy@172.17.0.9:2552] is JOINING, roles [dc-default] 2021-02-02 00:04:55.859 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Leader is moving node [akka://cloudstate-proxy@172.17.0.10:2552] to [Up] 2021-02-02 00:04:55.860 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Leader is moving node [akka://cloudstate-proxy@172.17.0.9:2552] to [Up] 2021-02-02 00:04:55.860 INFO akka.cluster.sbr.SplitBrainResolver - This node is not the leader any more and not responsible for taking SBR decisions. 2021-02-02 00:04:56.880 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - is no longer leader 2021-02-02 00:05:03.690 ERROR akka.cluster.sharding.DDataShardCoordinator - The ShardCoordinator was unable to update a distributed state within 'updating-state-timeout': 5000 millis (retrying). Perhaps the ShardRegion has not started on all active nodes yet? event=ShardRegionRegistered(Actor[akka://cloudstate-proxy@172.17.0.9:2552/system/sharding/eventsourced-shopping-cart#1207442416]) 2021-02-02 00:05:09.125 WARN akka.actor.ActorSystemImpl - Failed to load JFR flight recorder, falling back to noop. Exception: java.lang.ClassNotFoundException: akka.cluster.sharding.internal.jfr.JFRShardingFlightRecorder 2021-02-02 00:05:09.130 INFO com.datastax.driver.core.GuavaCompatibility - Detected Guava >= 19 in the classpath, using modern compatibility layer 2021-02-02 00:05:09.139 INFO com.datastax.driver.core - DataStax Java driver 3.7.2 for Apache Cassandra 2021-02-02 00:05:09.140 INFO com.datastax.driver.core.Native - Could not load JNR C Library, native system calls through this library will not be available (set this logger level to DEBUG to see the full stack trace). 2021-02-02 00:05:09.140 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps. 2021-02-02 00:05:09.140 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps. 2021-02-02 00:05:09.142 INFO com.datastax.driver.core.NettyUtil - Did not find Netty's native epoll transport in the classpath, defaulting to NIO. 2021-02-02 00:05:09.143 INFO com.datastax.driver.core.Native - Could not load JNR POSIX Library, native system calls through this library will not be available (set this logger level to DEBUG to see the full stack trace). 2021-02-02 00:05:09.144 INFO com.datastax.driver.core.utils.UUIDs - PID obtained through JMX: 1 2021-02-02 00:05:09.144 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps. 2021-02-02 00:05:09.369 INFO com.datastax.driver.core.policies.DCAwareRoundRobinPolicy - Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor) 2021-02-02 00:05:09.369 INFO com.datastax.driver.core.Cluster - New Cassandra host cassandra.default.svc.cluster.local/10.107.191.162:9042 added 2021-02-02 00:05:09.409 INFO com.datastax.driver.core.policies.DCAwareRoundRobinPolicy - Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor) 2021-02-02 00:05:09.409 INFO com.datastax.driver.core.Cluster - New Cassandra host cassandra.default.svc.cluster.local/10.107.191.162:9042 added 2021-02-02 00:05:22.470 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps. 2021-02-02 00:05:22.590 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps. 2021-02-02 00:05:29.149 ERROR io.cloudstate.proxy.eventsourced.EventSourcedEntity - Persistence failure when replaying events for persistenceId [eventsourced-shopping-cart|test1]. Last known sequence number [0] akka.pattern.CircuitBreaker$$anon$13: Circuit Breaker Timed out. 2021-02-02 00:05:29.150 INFO akka.actor.LocalActorRef - Message [io.cloudstate.proxy.entity.EntityCommand] from Actor[akka://cloudstate-proxy@172.17.0.9:2552/temp/eventsourced-shopping-cart$a] to Actor[akka://cloudstate-proxy/system/sharding/eventsourced-shopping-cart/87/test1/entity#-425630766] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/system/sharding/eventsourced-shopping-cart/87/test1/entity#-425630766] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 2021-02-02 00:04:54.928 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2021-02-02 00:04:54.932 WARN akka.actor.ActorSystemImpl - Failed to load JFR flight recorder, falling back to noop. Exception: java.lang.ClassNotFoundException: akka.remote.artery.jfr.JFRRemotingFlightRecorder 2021-02-02 00:04:54.949 INFO akka.remote.artery.tcp.ArteryTcpTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cloudstate-proxy@172.17.0.9:2552] with UID [-2365786287080537531] 2021-02-02 00:04:54.949 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.9:2552] - Starting up, Akka version [2.6.9] ... 2021-02-02 00:04:54.950 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.9:2552] - Started up successfully 2021-02-02 00:04:54.950 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.9:2552] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 2021-02-02 00:04:54.951 INFO akka.cluster.sbr.SplitBrainResolver - SBR started. Config: stableAfter: 20000 ms, strategy: KeepMajority, selfUniqueAddress: UniqueAddress(akka://cloudstate-proxy@172.17.0.9:2552,-2365786287080537531), selfDc: default 2021-02-02 00:04:54.966 INFO akka.actor.ActorSystemImpl - Starting Cloudstate Proxy version [0.6.0-5-f0446998] running on [Substrate VM GraalVM 20.3.1 Java 11] 2021-02-02 00:04:54.967 INFO akka.actor.ActorSystemImpl - Registering SIGTERM handler... 2021-02-02 00:04:54.967 INFO akka.management.internal.HealthChecksImpl - Loading readiness checks [(cloudstate,io.cloudstate.proxy.HealthCheckReady)] 2021-02-02 00:04:54.967 INFO akka.management.internal.HealthChecksImpl - Loading liveness checks [(cloudstate,io.cloudstate.proxy.HealthCheckLive)] 2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Binding Akka Management (HTTP) endpoint to: 172.17.0.9:8558 2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for ClusterBootstrap 2021-02-02 00:04:54.969 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Using self contact point address: http://172.17.0.9:8558 2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for HealthCheckRoutes 2021-02-02 00:04:54.975 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Initiating bootstrap procedure using kubernetes-api method... 2021-02-02 00:04:54.975 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Bootstrap using `akka.discovery` method: kubernetes-api 2021-02-02 00:04:54.975 INFO akka.management.scaladsl.AkkaManagement - Bound Akka Management (HTTP) endpoint to: 172.17.0.9:8558 2021-02-02 00:04:54.978 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Locating service members. Using discovery [akka.discovery.kubernetes.KubernetesApiServiceDiscovery], join decider [akka.management.cluster.bootstrap.LowestAddressJoinDecider], scheme [http] 2021-02-02 00:04:54.979 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))] 2021-02-02 00:04:54.979 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None] 2021-02-02 00:04:55.001 INFO akka.actor.ActorSystemImpl - Prometheus exporter started on /0.0.0.0:9099 2021-02-02 00:04:55.065 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8))], filtered to [172-17-0-8.default.pod.cluster.local:0] 2021-02-02 00:04:55.066 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at io.grpc.Status.asRuntimeException(Status.java:533) at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426) at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:834) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192) Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080 Caused by: java.net.ConnectException: Connection refused at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0) at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:834) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192) 2021-02-02 00:04:55.086 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Contact point [akka://cloudstate-proxy@172.17.0.8:2552] returned [1] seed-nodes [akka://cloudstate-proxy@172.17.0.8:2552] 2021-02-02 00:04:55.087 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Joining [akka://cloudstate-proxy@172.17.0.9:2552] to existing cluster [akka://cloudstate-proxy@172.17.0.8:2552] 2021-02-02 00:04:55.087 INFO akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [akka.management.cluster.bootstrap.internal.BootstrapCoordinator$Protocol$BootstrappingCompleted$] from Actor[akka://cloudstate-proxy/system/bootstrapCoordinator#15500066] to Actor[akka://cloudstate-proxy/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 2021-02-02 00:04:55.378 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.9:2552] - Received InitJoinAck message from [Actor[akka://cloudstate-proxy@172.17.0.8:2552/system/cluster/core/daemon#-754755589]] to [akka://cloudstate-proxy@172.17.0.9:2552] 2021-02-02 00:04:55.380 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.9:2552] - Welcome from [akka://cloudstate-proxy@172.17.0.8:2552] 2021-02-02 00:04:56.179 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at io.grpc.Status.asRuntimeException(Status.java:533) at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426) at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:834) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192) Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080 Caused by: java.net.ConnectException: Connection refused at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0) at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:834) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192) 2021-02-02 00:04:58.583 INFO akka.management.internal.HealthChecksImpl - Check [io.cloudstate.proxy.HealthCheckReady] not ok 2021-02-02 00:04:58.672 INFO io.cloudstate.proxy.EntityDiscoveryManager - Received EntitySpec from user function with info: ServiceInfo(,,OpenJDK Runtime Environment 11.0.10+9,cloudstate-java-support,0.6.0-4-f0446998,0,2,UnknownFieldSet(Map())) 2021-02-02 00:04:58.672 DEBUG io.cloudstate.proxy.eventsourced.EventSourcedSupportFactory - Starting EventSourcedEntity for eventsourced-shopping-cart 2021-02-02 00:04:58.674 INFO akka.cluster.singleton.ClusterSingletonManager - ClusterSingletonManager state change [Start -> Younger] 2021-02-02 00:04:58.674 INFO EventingManager - Eventing support turned off in configuration 2021-02-02 00:04:58.674 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/add),*,,Vector(),UnknownFieldSet(Map()))] 2021-02-02 00:04:58.675 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/{product_id}/remove),,,Vector(),UnknownFieldSet(Map()))] 2021-02-02 00:04:58.675 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Get(/carts/{user_id}),,,Vector(HttpRule(,Get(/carts/{user_id}/items),,items,Vector(),UnknownFieldSet(Map()))),UnknownFieldSet(Map()))] 2021-02-02 00:04:58.676 DEBUG io.cloudstate.proxy.EntityDiscoveryManager - Starting gRPC proxy 2021-02-02 00:04:58.680 INFO io.cloudstate.proxy.EntityDiscoveryManager - CloudState proxy online at /0.0.0.0:8013 2021-02-02 00:05:09.122 DEBUG io.cloudstate.proxy.HttpApi$HttpEndpoint - Received HTTP request [/carts/test1] for [com.example.shoppingcart.ShoppingCart.GetCart] * Request Field: [com.example.shoppingcart.GetShoppingCart.user_id] = [test1] 2021-02-02 00:05:09.122 DEBUG io.cloudstate.proxy.Serve$ - Received gRPC request [/com.example.shoppingcart.ShoppingCart/GetCart] 2021-02-02 00:05:29.134 INFO akka.actor.ActorSystemImpl - Request timeout encountered for request [GET /carts/test1 Empty] === User function logs === 2021-02-02 00:04:56.069 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$AddLineItem 2021-02-02 00:04:56.075 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.google.protobuf.Empty 2021-02-02 00:04:56.077 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$RemoveLineItem 2021-02-02 00:04:56.079 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$GetShoppingCart 2021-02-02 00:04:56.081 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$Cart [INFO] [02/02/2021 00:04:58.893] [StatefulService-akka.actor.default-dispatcher-6] [akka.actor.ActorSystemImpl(StatefulService)] Received discovery call from [cloudstate-proxy-core 0.6.0-5-f0446998] supporting Cloudstate protocol 0.2 2021-02-02 00:04:47.408 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$AddLineItem 2021-02-02 00:04:47.411 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.google.protobuf.Empty 2021-02-02 00:04:47.413 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$RemoveLineItem 2021-02-02 00:04:47.414 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$GetShoppingCart 2021-02-02 00:04:47.416 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$Cart [INFO] [02/02/2021 00:04:50.179] [StatefulService-akka.actor.default-dispatcher-6] [akka.actor.ActorSystemImpl(StatefulService)] Received discovery call from [cloudstate-proxy-core 0.6.0-5-f0446998] supporting Cloudstate protocol 0.2 [ERROR] [02/02/2021 00:05:29.161] [StatefulService-akka.actor.default-dispatcher-10] [EventSourcedImpl] Terminating entity due to unexpected failure io.cloudstate.javasupport.impl.EntityExceptions$EntityException: Protocol error: Expected Init message at io.cloudstate.javasupport.impl.EntityExceptions$ProtocolException$.apply(EntityExceptions.scala:46) at io.cloudstate.javasupport.impl.eventsourced.EventSourcedImpl.$anonfun$handle$1(EventSourcedImpl.scala:125) at akka.stream.impl.fusing.Map$$anon$1.onPush(Ops.scala:53) at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:541) at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:527) at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:390) at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:625) at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:502) at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:600) at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:769) at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$shortCircuitBatch(ActorGraphInterpreter.scala:759) at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:785) at akka.actor.Actor.aroundReceive(Actor.scala:537) at akka.actor.Actor.aroundReceive$(Actor.scala:535) at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:691) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577) at akka.actor.ActorCell.invoke(ActorCell.scala:547) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) at akka.dispatch.Mailbox.run(Mailbox.scala:231) at akka.dispatch.Mailbox.exec(Mailbox.scala:243) at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) 2021-02-02 00:04:55.907 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$AddLineItem 2021-02-02 00:04:55.911 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.google.protobuf.Empty 2021-02-02 00:04:55.913 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$RemoveLineItem 2021-02-02 00:04:55.915 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$GetShoppingCart 2021-02-02 00:04:55.916 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$Cart [INFO] [02/02/2021 00:04:58.609] [StatefulService-akka.actor.default-dispatcher-5] [akka.actor.ActorSystemImpl(StatefulService)] Received discovery call from [cloudstate-proxy-core 0.6.0-5-f0446998] supporting Cloudstate protocol 0.2 Exited with code exit status 1 CircleCI received exit code 1 ```

[sleipnir]

Judging by these logs it seems to me that the proxy is not activating the entity first.

[pvlugter]

Yeah, it's not managing to initialise the entity first. I think the timeout on replaying events will be the issue:

ERROR io.cloudstate.proxy.eventsourced.EventSourcedEntity - Persistence failure when replaying events for persistenceId [eventsourced-shopping-cart|test1]. Last known sequence number [0]
akka.pattern.CircuitBreaker$$anon$13: Circuit Breaker Timed out.

The user function error, that it expected an init message, is not as useful without saying what it received instead.