kamus:controller-0.4.4.1 readiness and liveness probe is restarting quite alot

[ocofaigh]

Describe the bug kamus:controller-0.4.4.1 readiness and liveness probe is restarting quite alot. Log snippet:

$ kubectl logs kamus-controller-b9548dc4b-qgd6d --previous
{"Timestamp":"2020-09-18T21:15:53.5642936+00:00","Level":"Information","MessageTemplate":"Starting watch for KamusSecret V1Alpha events","Properties":{"SourceContext":"CustomResourceDescriptorController.HostedServices.V1AlphaController"}}
Hosting environment: Production
Content root path: /home/dotnet/app
Now listening on: http://[::]:9999
Application started. Press Ctrl+C to shut down.
{"Timestamp":"2020-09-18T21:15:54.5083521+00:00","Level":"Information","MessageTemplate":"Handling event of type {type}. KamusSecret {name} in namespace {namespace}","Properties":{"type":"Added","name":"argocd-secret-test","namespace":"argocd","SourceContext":"CustomResourceDescriptorController.HostedServices.V1AlphaController"}}
{"Timestamp":"2020-09-18T21:15:55.5641789+00:00","Level":"Information","MessageTemplate":"Handling event of type {type}. KamusSecret {name} in namespace {namespace}","Properties":{"type":"Added","name":"star-cpsre-dev-cloud-ibm-com","namespace":"default","SourceContext":"CustomResourceDescriptorController.HostedServices.V1AlphaController"}}
{"Timestamp":"2020-09-18T21:15:56.0038841+00:00","Level":"Error","MessageTemplate":"Error while handling KamusSecret event of type {eventType}, for KamusSecret {name} on namespace {namespace}","Exception":"Microsoft.Rest.HttpOperationException: Operation returned an invalid status code 'Conflict'\n   at k8s.Kubernetes.CreateNamespacedSecretWithHttpMessagesAsync(V1Secret body, String namespaceParameter, String dryRun, String fieldManager, String pretty, Dictionary`2 customHeaders, CancellationToken cancellationToken)\n   at k8s.KubernetesExtensions.CreateNamespacedSecretAsync(IKubernetes operations, V1Secret body, String namespaceParameter, String dryRun, String fieldManager, String pretty, CancellationToken cancellationToken)\n   at CustomResourceDescriptorController.HostedServices.V1AlphaController.HandleAdd(KamusSecret kamusSecret, Boolean isUpdate) in /app/crd-controller/HostedServices/V1AlphaController.cs:line 172\n   at CustomResourceDescriptorController.HostedServices.V1AlphaController.HandleEvent(WatchEventType event, KamusSecret kamusSecret) in /app/crd-controller/HostedServices/V1AlphaController.cs:line 92","Properties":{"eventType":"Added","name":"star-cpsre-dev-cloud-ibm-com","namespace":"default","SourceContext":"CustomResourceDescriptorController.HostedServices.V1AlphaController"}}
{"Timestamp":"2020-09-18T21:15:55.8910508+00:00","Level":"Error","MessageTemplate":"Error while handling KamusSecret event of type {eventType}, for KamusSecret {name} on namespace {namespace}","Exception":"Microsoft.Rest.HttpOperationException: Operation returned an invalid status code 'Conflict'\n   at k8s.Kubernetes.CreateNamespacedSecretWithHttpMessagesAsync(V1Secret body, String namespaceParameter, String dryRun, String fieldManager, String pretty, Dictionary`2 customHeaders, CancellationToken cancellationToken)\n   at k8s.KubernetesExtensions.CreateNamespacedSecretAsync(IKubernetes operations, V1Secret body, String namespaceParameter, String dryRun, String fieldManager, String pretty, CancellationToken cancellationToken)\n   at CustomResourceDescriptorController.HostedServices.V1AlphaController.HandleAdd(KamusSecret kamusSecret, Boolean isUpdate) in /app/crd-controller/HostedServices/V1AlphaController.cs:line 172\n   at CustomResourceDescriptorController.HostedServices.V1AlphaController.HandleEvent(WatchEventType event, KamusSecret kamusSecret) in /app/crd-controller/HostedServices/V1AlphaController.cs:line 92","Properties":{"eventType":"Added","name":"argocd-secret-test","namespace":"argocd","SourceContext":"CustomResourceDescriptorController.HostedServices.V1AlphaController"}}
{"Timestamp":"2020-09-18T21:25:54.1083798+00:00","Level":"Error","MessageTemplate":"Unexpected error occured while watching KamusSecret events","Exception":"System.IO.IOException: The server returned an invalid or unrecognized response.\n   at System.Net.Http.HttpConnection.FillAsync()\n   at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.ReadAsyncCore(Memory`1 buffer, CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnection.HttpContentReadStream.Read(Byte[] buffer, Int32 offset, Int32 count)\n   at System.IO.StreamReader.ReadBuffer()\n   at System.IO.StreamReader.ReadLine()\n   at System.Threading.Tasks.Task`1.InnerInvoke()\n   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)\n--- End of stack trace from previous location where exception was thrown ---\n   at k8s.Watcher`1.WatcherLoop(CancellationToken cancellationToken)","Properties":{"SourceContext":"CustomResourceDescriptorController.HostedServices.V1AlphaController"}}
Application is shutting down...

$ kubectl get pods | grep kamus
kamus-controller-b9548dc4b-qgd6d         1/1     Running      19252      134d
kamus-decryptor-5cdf4ff7f8-bwgcl         1/1     Running      2          134d
kamus-decryptor-5cdf4ff7f8-t7znr         1/1     Running      3          134d
kamus-encryptor-6c4675d5cd-59bj4         1/1     Running      0          134d
kamus-encryptor-6c4675d5cd-blp42         1/1     Running      0          134d

I see that the latest version is 0.8.0.0. Would it be worth upgrading to the latest? And if so, what is the upgrade path? I did not use helm to install. I used the following commands:

#!/usr/bin/env bash
helm init -c
helm repo add soluto https://charts.soluto.io
key=$(openssl rand -base64 32 | tr -d '\n')
SOURCE=$(pwd)
mkdir tmp
cd tmp
helm fetch soluto/kamus --untar && helm template kamus --set keyManagement.AES.key=$key > kamus-manifest-AES.yaml
kubectl apply -f kamus-manifest-AES.yaml
cd $SOURCE
rm -rf tmp

Versions used kamus:controller-0.4.4.1 kubernetes v1.16.14

[shaikatz]

Hi, the controller being restarted every 60 minutes by definition, since this the watch timeout that was defined. We might increase that to be infinite in the future, but we didn't get to test it yet. It should not be a problem, and you can just safely ignore it in the meantime as long as you get the correct functionality.

[ocofaigh]

Thanks @shaikatz - we will ignore the restarts. How about the upgrade process considering I used the script above to deploy and not helm directly? Would you suggest upgrading, or is the version we are on stable enough?

[shaikatz]

As long as you are using AES I don't think there are major improvements in the newer versions. Your version is stable, and you can keep using it. In the future, if you want to upgrade, the process should be easy, just running the same command you mentioned again. Just make sure you don't generate the AES key again, but taking the existing one.

[ocofaigh]

Thanks @shaikatz - I'll close the issue now