Closed pieterv-icloud-com closed 1 year ago
odubajDT
commented
1 year ago Hello @pieterv-icloud-com, thank you for submitting the ticket! May I ask you to also show your KeptnApp and also KeptnAppVersion?
Also I would have another question, are you using "automatic app discovery" feature to create your KeptnApp or ar you creating it manually?
Thank you!
thisthat
commented
1 year ago maybe it could be similar to https://github.com/keptn/lifecycle-toolkit/issues/1878
pieterv-icloud-com
commented
1 year ago Hi, we are using the automatic app discovery feature
Here are the details you requested
KeptnApp:
Name: myapp
Namespace: myapp
Labels: app.kubernetes.io/managed-by=klt
Annotations: <none>
API Version: lifecycle.keptn.sh/v1alpha3
Kind: KeptnApp
Metadata:
Creation Timestamp: 2023-09-11T09:50:12Z
Generation: 2
Resource Version: 8157
UID: 140b4094-532e-48f2-8ef0-51ca0f4ef815
Spec:
Revision: 1
Version: 0.1.7
Workloads:
Name: node-web-app
Version: 0.1.7
Status:
Current Version: 0.1.7
Events: <none>KeptnAppVersion:
Name: myapp-0.1.7-d4735e3a
Namespace: myapp
Labels: <none>
Annotations: traceparent: 00-08572a00bb3d70a096a0402894a8dea4-5832f6e6e08c5b6f-01
API Version: lifecycle.keptn.sh/v1alpha3
Kind: KeptnAppVersion
Metadata:
Creation Timestamp: 2023-09-11T10:10:55Z
Generation: 1
Owner References:
API Version: lifecycle.keptn.sh/v1alpha3
Block Owner Deletion: true
Controller: true
Kind: KeptnApp
Name: myapp
UID: 140b4094-532e-48f2-8ef0-51ca0f4ef815
Resource Version: 8203
UID: d2d66aea-6e3a-4073-8531-38b1500d6817
Spec:
App Name: myapp
Previous Version: 0.1.6
Revision: 1
Trace Id:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-cc6a684f3e4db67f-01
Version: 0.1.7
Workloads:
Name: node-web-app
Version: 0.1.7
Status:
Current Phase: AppDeploy
Phase Trace I Ds:
:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-70d6172dc287ae22-01
App Deploy:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-f012625e51f019a1-01
App Pre Deploy Evaluations:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-2c139a1177d1d822-01
App Pre Deploy Tasks:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-4f4540e8587d2192-01
Post Deployment Evaluation Status: Pending
Post Deployment Status: Pending
Pre Deployment Evaluation Status: Succeeded
Pre Deployment Status: Succeeded
Start Time: 2023-09-11T10:10:55Z
Status: Progressing
Workload Overall Status: Progressing
Workload Status:
Status: Progressing
Workload:
Name: node-web-app
Version: 0.1.7
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AppPreDeployTasksStarted 5m49s keptnappversion-controller App Pre-Deployment Tasks: has started / Namespace: myapp, Name: myapp-0.1.7-d4735e3a, Version: 0.1.7
Normal AppPreDeployTasksFinished 5m49s keptnappversion-controller App Pre-Deployment Tasks: has finished / Namespace: myapp, Name: myapp-0.1.7-d4735e3a, Version: 0.1.7
Normal AppPreDeployEvaluationsStarted 5m49s keptnappversion-controller App Pre-Deployment Evaluations: has started / Namespace: myapp, Name: myapp-0.1.7-d4735e3a, Version: 0.1.7
Normal AppPreDeployEvaluationsFinished 5m49s keptnappversion-controller App Pre-Deployment Evaluations: has finished / Namespace: myapp, Name: myapp-0.1.7-d4735e3a, Version: 0.1.7
Normal AppDeployStarted 5m49s keptnappversion-controller App Deployment: has started / Namespace: myapp, Name: myapp-0.1.7-d4735e3a, Version: 0.1.7KeptnWorkloadInstance:
Name: myapp-node-web-app-0.1.7
Namespace: myapp
Labels: <none>
Annotations: traceparent: 00-0aec4a65efc7fa2c877f5db90393e3d2-4b9bd1e00a787749-01
API Version: lifecycle.keptn.sh/v1alpha3
Kind: KeptnWorkloadInstance
Metadata:
Creation Timestamp: 2023-09-11T10:10:15Z
Generation: 2
Owner References:
API Version: lifecycle.keptn.sh/v1alpha3
Block Owner Deletion: true
Controller: true
Kind: KeptnWorkload
Name: myapp-node-web-app
UID: a0fc1484-dbeb-4d6f-93c2-5e70de00ac37
Resource Version: 8247
UID: 45a54d53-8987-4e5c-ae97-178c15d1862f
Spec:
App: myapp
Post Deployment Tasks:
post-deployment
Pre Deployment Evaluations:
cpu-throttling
Pre Deployment Tasks:
pre-deployment
Previous Version: 0.1.6
Resource Reference:
Kind: ReplicaSet
Name: node-web-app-6df48c876f
UID: b74a45bd-80df-4df8-a036-a0e29c5efb8e
Trace Id:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-f012625e51f019a1-01
Version: 0.1.7
Workload Name: myapp-node-web-app
Status:
Current Phase: WorkloadPreDeployEvaluations
Deployment Status: Pending
Phase Trace I Ds:
:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-13105f8a94568253-01
Workload Pre Deploy Evaluations:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-e011eb699e30a0c3-01
Workload Pre Deploy Tasks:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-590b8379ed9d3fd5-01
Post Deployment Evaluation Status: Pending
Post Deployment Status: Pending
Pre Deployment Evaluation Status: Pending
Pre Deployment Evaluation Task Status:
Definition Name: cpu-throttling
Name: pre-eval-cpu-throttling-89288
Start Time: 2023-09-11T10:11:07Z
Status: Pending
Pre Deployment Status: Succeeded
Pre Deployment Task Status:
Definition Name: pre-deployment
End Time: 2023-09-11T10:11:07Z
Name: pre-pre-deployment-60012
Start Time: 2023-09-11T10:10:56Z
Status: Succeeded
Start Time: 2023-09-11T10:10:57Z
Status: Progressing
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning AppPreDeployEvaluationsAppVersionNotFound 9m11s (x13 over 9m32s) keptnworkloadinstance-controller App Pre-Deployment Evaluations: has failed since app could not be found / Namespace: myapp, Name: myapp-node-web-app-0.1.7, Version: 0.1.7
Normal WorkloadPreDeployTasksStarted 8m51s keptnworkloadinstance-controller Workload Pre-Deployment Tasks: has started / Namespace: myapp, Name: myapp-node-web-app-0.1.7, Version: 0.1.7
Normal ReconcileTaskStatusChanged 8m51s keptnworkloadinstance-controller Reconcile Task: task status changed from to Pending / Namespace: myapp, Name: myapp-node-web-app-0.1.7, Version: 0.1.7
Warning WorkloadPreDeployTasksReconcileError 8m50s keptnworkloadinstance-controller Workload Pre-Deployment Tasks: could not get reconciled / Namespace: myapp, Name: myapp-node-web-app-0.1.7, Version: 0.1.7
Normal WorkloadPreDeployTasksFinished 8m40s keptnworkloadinstance-controller Workload Pre-Deployment Tasks: has finished / Namespace: myapp, Name: myapp-node-web-app-0.1.7, Version: 0.1.7
Normal WorkloadPreDeployEvaluationsStarted 8m40s keptnworkloadinstance-controller Workload Pre-Deployment Evaluations: has started / Namespace: myapp, Name: myapp-node-web-app-0.1.7, Version: 0.1.7
Normal ReconcileEvaluationStatusChanged 8m40s keptnworkloadinstance-controller Reconcile Evaluation: evaluation status changed from to Pending / Namespace: myapp, Name: myapp-node-web-app-0.1.7, Version: 0.1.7Thank you very much for the info! I will go through it and try to find the problem. I will get back into you as soon as I possible
pieterv-icloud-com
commented
1 year ago Let me know if there is anything further you require
odubajDT
commented
1 year ago Can you please show also all the deprecated KeptnAppVersions in the myapp namespace? Basically executing kubectl describe on every KeptnAppVersion in this namepace.
Thank you
pieterv-icloud-com
commented
1 year ago Sure here they are:
Name: myapp-0.1.6-6b86b273
Namespace: myapp
Labels: <none>
Annotations: <none>
API Version: lifecycle.keptn.sh/v1alpha3
Kind: KeptnAppVersion
Metadata:
Creation Timestamp: 2023-09-11T09:50:12Z
Generation: 1
Owner References:
API Version: lifecycle.keptn.sh/v1alpha3
Block Owner Deletion: true
Controller: true
Kind: KeptnApp
Name: myapp
UID: 140b4094-532e-48f2-8ef0-51ca0f4ef815
Resource Version: 4283
UID: fd400122-565c-4656-b76e-62b61d612857
Spec:
App Name: myapp
Revision: 1
Version: 0.1.6
Workloads:
Name: node-web-app
Version: 0.1.6
Status:
Current Phase: AppDeploy
End Time: 2023-09-11T09:55:34Z
Phase Trace I Ds:
:
App Deploy:
App Pre Deploy Evaluations:
App Pre Deploy Tasks:
Post Deployment Evaluation Status: Deprecated
Post Deployment Status: Deprecated
Pre Deployment Evaluation Status: Succeeded
Pre Deployment Status: Succeeded
Start Time: 2023-09-11T09:50:12Z
Status: Failed
Workload Overall Status: Failed
Workload Status:
Status: Failed
Workload:
Name: node-web-app
Version: 0.1.6
Events: <none>
Name: myapp-0.1.7-d4735e3a
Namespace: myapp
Labels: <none>
Annotations: traceparent: 00-08572a00bb3d70a096a0402894a8dea4-5832f6e6e08c5b6f-01
API Version: lifecycle.keptn.sh/v1alpha3
Kind: KeptnAppVersion
Metadata:
Creation Timestamp: 2023-09-11T10:10:55Z
Generation: 1
Owner References:
API Version: lifecycle.keptn.sh/v1alpha3
Block Owner Deletion: true
Controller: true
Kind: KeptnApp
Name: myapp
UID: 140b4094-532e-48f2-8ef0-51ca0f4ef815
Resource Version: 8203
UID: d2d66aea-6e3a-4073-8531-38b1500d6817
Spec:
App Name: myapp
Previous Version: 0.1.6
Revision: 1
Trace Id:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-cc6a684f3e4db67f-01
Version: 0.1.7
Workloads:
Name: node-web-app
Version: 0.1.7
Status:
Current Phase: AppDeploy
Phase Trace I Ds:
:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-70d6172dc287ae22-01
App Deploy:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-f012625e51f019a1-01
App Pre Deploy Evaluations:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-2c139a1177d1d822-01
App Pre Deploy Tasks:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-4f4540e8587d2192-01
Post Deployment Evaluation Status: Pending
Post Deployment Status: Pending
Pre Deployment Evaluation Status: Succeeded
Pre Deployment Status: Succeeded
Start Time: 2023-09-11T10:10:55Z
Status: Progressing
Workload Overall Status: Progressing
Workload Status:
Status: Progressing
Workload:
Name: node-web-app
Version: 0.1.7
Events: <none>
Name: myapp-0.1.8-4e074085
Namespace: myapp
Labels: <none>
Annotations: traceparent: 00-9f2b87461b1f0cc5e85523c2fda2b71a-67fd618323d9e7b2-01
API Version: lifecycle.keptn.sh/v1alpha3
Kind: KeptnAppVersion
Metadata:
Creation Timestamp: 2023-09-11T10:15:51Z
Generation: 1
Owner References:
API Version: lifecycle.keptn.sh/v1alpha3
Block Owner Deletion: true
Controller: true
Kind: KeptnApp
Name: myapp
UID: 140b4094-532e-48f2-8ef0-51ca0f4ef815
Resource Version: 9341
UID: 2307a858-0031-42d4-aba0-0ccd57bb99be
Spec:
App Name: myapp
Previous Version: 0.1.7
Revision: 1
Trace Id:
Traceparent: 00-f0af258be699a59eaa58c8708891243d-aeea3ac171762765-01
Version: 0.1.8
Workloads:
Name: node-web-app
Version: 0.1.8
Status:
Current Phase: AppDeploy
Phase Trace I Ds:
:
Traceparent: 00-f0af258be699a59eaa58c8708891243d-3909afa2d32acbe2-01
App Deploy:
Traceparent: 00-f0af258be699a59eaa58c8708891243d-49cb847fc2be461b-01
App Pre Deploy Evaluations:
Traceparent: 00-f0af258be699a59eaa58c8708891243d-ed9a78dbc6c39eb2-01
App Pre Deploy Tasks:
Traceparent: 00-f0af258be699a59eaa58c8708891243d-f48afa2810a6b92f-01
Post Deployment Evaluation Status: Pending
Post Deployment Status: Pending
Pre Deployment Evaluation Status: Succeeded
Pre Deployment Status: Succeeded
Start Time: 2023-09-11T10:15:51Z
Status: Progressing
Workload Overall Status: Progressing
Workload Status:
Status: Progressing
Workload:
Name: node-web-app
Version: 0.1.8
Events: <none>
This is interesting, you have 2 KeptnAppVersions active in your system. This causes problem with finding the latest KeptnAppVersion for the WorkloadInstance. The question is, why KeptnAppVersion myapp-0.1.7-d4735e3a was not deprecated when myapp-0.1.8-4e074085 was created...
Can you please show me all the WorkloadInstances in the myapp namespace?
pieterv-icloud-com
commented
1 year ago Here goes:
Name: myapp-node-web-app-0.1.6
Namespace: myapp
Labels: <none>
Annotations: traceparent: 00-0aec4a65efc7fa2c877f5db90393e3d2-e9beba15b91384ea-01
API Version: lifecycle.keptn.sh/v1alpha3
Kind: KeptnWorkloadInstance
Metadata:
Creation Timestamp: 2023-09-11T09:57:57Z
Generation: 1
Owner References:
API Version: lifecycle.keptn.sh/v1alpha3
Block Owner Deletion: true
Controller: true
Kind: KeptnWorkload
Name: myapp-node-web-app
UID: a0fc1484-dbeb-4d6f-93c2-5e70de00ac37
Resource Version: 7850
UID: d3ce591b-3b2a-40da-9c6d-47296767686a
Spec:
App: myapp
Post Deployment Tasks:
post-deployment
Pre Deployment Evaluations:
cpu-throttling
Pre Deployment Tasks:
pre-deployment
Resource Reference:
Kind: ReplicaSet
Name: node-web-app-cbcd86954
UID: e482f4e1-3418-49c2-9e8d-8ac708f5731f
Version: 0.1.6
Workload Name: myapp-node-web-app
Status:
Current Phase: WorkloadPreDeployEvaluations
Deployment Status: Pending
Phase Trace I Ds:
:
Workload Pre Deploy Evaluations:
Workload Pre Deploy Tasks:
Traceparent: 00-47e1f8bfc94dc403166991605b930baf-519834cda1978611-01
Post Deployment Evaluation Status: Pending
Post Deployment Status: Pending
Pre Deployment Evaluation Status: Pending
Pre Deployment Evaluation Task Status:
Definition Name: cpu-throttling
Name: pre-eval-cpu-throttling-97257
Start Time: 2023-09-11T09:58:18Z
Status: Pending
Pre Deployment Status: Succeeded
Pre Deployment Task Status:
Definition Name: pre-deployment
End Time: 2023-09-11T09:58:18Z
Name: pre-pre-deployment-97954
Start Time: 2023-09-11T09:57:57Z
Status: Succeeded
Start Time: 2023-09-11T09:58:02Z
Status: Progressing
Events: <none>
Name: myapp-node-web-app-0.1.7
Namespace: myapp
Labels: <none>
Annotations: traceparent: 00-0aec4a65efc7fa2c877f5db90393e3d2-4b9bd1e00a787749-01
API Version: lifecycle.keptn.sh/v1alpha3
Kind: KeptnWorkloadInstance
Metadata:
Creation Timestamp: 2023-09-11T10:10:15Z
Generation: 2
Owner References:
API Version: lifecycle.keptn.sh/v1alpha3
Block Owner Deletion: true
Controller: true
Kind: KeptnWorkload
Name: myapp-node-web-app
UID: a0fc1484-dbeb-4d6f-93c2-5e70de00ac37
Resource Version: 8247
UID: 45a54d53-8987-4e5c-ae97-178c15d1862f
Spec:
App: myapp
Post Deployment Tasks:
post-deployment
Pre Deployment Evaluations:
cpu-throttling
Pre Deployment Tasks:
pre-deployment
Previous Version: 0.1.6
Resource Reference:
Kind: ReplicaSet
Name: node-web-app-6df48c876f
UID: b74a45bd-80df-4df8-a036-a0e29c5efb8e
Trace Id:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-f012625e51f019a1-01
Version: 0.1.7
Workload Name: myapp-node-web-app
Status:
Current Phase: WorkloadPreDeployEvaluations
Deployment Status: Pending
Phase Trace I Ds:
:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-13105f8a94568253-01
Workload Pre Deploy Evaluations:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-e011eb699e30a0c3-01
Workload Pre Deploy Tasks:
Traceparent: 00-e87ec08474f4e5e016c8cf49253fbccb-590b8379ed9d3fd5-01
Post Deployment Evaluation Status: Pending
Post Deployment Status: Pending
Pre Deployment Evaluation Status: Pending
Pre Deployment Evaluation Task Status:
Definition Name: cpu-throttling
Name: pre-eval-cpu-throttling-89288
Start Time: 2023-09-11T10:11:07Z
Status: Pending
Pre Deployment Status: Succeeded
Pre Deployment Task Status:
Definition Name: pre-deployment
End Time: 2023-09-11T10:11:07Z
Name: pre-pre-deployment-60012
Start Time: 2023-09-11T10:10:56Z
Status: Succeeded
Start Time: 2023-09-11T10:10:57Z
Status: Progressing
Events: <none>
Name: myapp-node-web-app-0.1.8
Namespace: myapp
Labels: <none>
Annotations: traceparent: 00-0aec4a65efc7fa2c877f5db90393e3d2-74e21d7f509505e5-01
API Version: lifecycle.keptn.sh/v1alpha3
Kind: KeptnWorkloadInstance
Metadata:
Creation Timestamp: 2023-09-11T10:15:11Z
Generation: 2
Owner References:
API Version: lifecycle.keptn.sh/v1alpha3
Block Owner Deletion: true
Controller: true
Kind: KeptnWorkload
Name: myapp-node-web-app
UID: a0fc1484-dbeb-4d6f-93c2-5e70de00ac37
Resource Version: 9395
UID: 9546ec2d-d982-4812-be37-6324cabc8733
Spec:
App: myapp
Post Deployment Tasks:
post-deployment
Pre Deployment Evaluations:
cpu-throttling
Pre Deployment Tasks:
pre-deployment
Previous Version: 0.1.7
Resource Reference:
Kind: ReplicaSet
Name: node-web-app-6487dbbccb
UID: 28207fed-b78b-4a0f-9e25-198051e12864
Trace Id:
Traceparent: 00-f0af258be699a59eaa58c8708891243d-49cb847fc2be461b-01
Version: 0.1.8
Workload Name: myapp-node-web-app
Status:
Current Phase: WorkloadPreDeployEvaluations
Deployment Status: Pending
Phase Trace I Ds:
:
Traceparent: 00-f0af258be699a59eaa58c8708891243d-8f054ffec8688f16-01
Workload Pre Deploy Evaluations:
Traceparent: 00-f0af258be699a59eaa58c8708891243d-7de2207914550d3b-01
Workload Pre Deploy Tasks:
Traceparent: 00-f0af258be699a59eaa58c8708891243d-7412525ef1b6ee20-01
Post Deployment Evaluation Status: Pending
Post Deployment Status: Pending
Pre Deployment Evaluation Status: Pending
Pre Deployment Evaluation Task Status:
Definition Name: cpu-throttling
Name: pre-eval-cpu-throttling-47202
Start Time: 2023-09-11T10:16:07Z
Status: Pending
Pre Deployment Status: Succeeded
Pre Deployment Task Status:
Definition Name: pre-deployment
End Time: 2023-09-11T10:16:07Z
Name: pre-pre-deployment-75808
Start Time: 2023-09-11T10:15:52Z
Status: Succeeded
Start Time: 2023-09-11T10:15:52Z
Status: Progressing
Events: <none>
Something else I just tried is to remove these annotations from my deployment:
keptn.sh/pre-deployment-tasks: pre-deployment
keptn.sh/post-deployment-tasks: post-deployment
keptn.sh/pre-deployment-evaluations: cpu-throttlingAnd the workload deploys without any issue:
NAMESPACE NAME APPNAME VERSION PHASE
myapp myapp-0.1.6-6b86b273 myapp 0.1.6 AppDeploy
myapp myapp-0.1.7-d4735e3a myapp 0.1.7 AppDeploy
myapp myapp-0.1.8-4e074085 myapp 0.1.8 AppDeploy
myapp myapp-0.1.9-4b227777 myapp 0.1.9 Completed
myapp myapp-0.1.10-ef2d127d myapp 0.1.10
myapp myapp-0.1.10-ef2d127d myapp 0.1.10 AppPreDeployTasks
myapp myapp-0.1.10-ef2d127d myapp 0.1.10 AppPreDeployEvaluations
myapp myapp-0.1.10-ef2d127d myapp 0.1.10 AppDeploy
myapp myapp-0.1.10-ef2d127d myapp 0.1.10 AppDeploy
myapp myapp-0.1.10-ef2d127d myapp 0.1.10 AppDeploy
myapp myapp-0.1.10-ef2d127d myapp 0.1.10 AppDeploy
myapp myapp-0.1.10-ef2d127d myapp 0.1.10 AppPostDeployTasks
myapp myapp-0.1.10-ef2d127d myapp 0.1.10 AppPostDeployEvaluations
myapp myapp-0.1.10-ef2d127d myapp 0.1.10 Completedyes, the check which obviously is causing issues and blocking the deployment is part of evaluation logic. We need to discuss this in the team and as soon as we have a solution I will let you know
pieterv-icloud-com
commented
1 year ago I did a couple of deployments now and it seems the annotation where the deployment stays in a pending state is this one:
keptn.sh/pre-deployment-evaluations: cpu-throttling
odubajDT
commented
1 year ago Thank you for info!
If I understand it correctly, you were never able to finish the deployment of your application, right? The the cause of it is that you have always 2 or more active KeptnAppVersions in your system, which causes that the pre-evaluation cpu-throttling check is blocked and therefore it's stuck. The real question is why the pre-deployment check pre-deployment was not blocked and went through
Can you please describe how exactly did you apply your manifests and when and how the versions were updated? I am trying to reproduce the problem but cannot...
pieterv-icloud-com
commented
1 year ago The deployments work fine except if a pre-deployment evaluation is specified, then the deployment get stuck in a Pending state.
Please let me know if this information is too detailed.
I have a devcontainer with a minikube cluster installed in it.
The cluster is bootstrapped with ArgoCD.
We then install various components like:
Once everything is deployed I will instruct ArgoCD to deploy the node-web-app using a helm chart:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
annotations:
argocd-image-updater.argoproj.io/container.platforms: linux/amd64,linux/arm64
argocd-image-updater.argoproj.io/container.update-strategy: semver
argocd-image-updater.argoproj.io/dex.helm.image-name: container.image.repository
argocd-image-updater.argoproj.io/dex.helm.image-tag: container.image.tag
argocd-image-updater.argoproj.io/image-list: container=localhost:5000/node-web-app
argocd-image-updater.argoproj.io/write-back-method: argocd
deimos.io/environment-development: deploy
deimos.io/environment-local: deploy
deimos.io/environment-production: deploy
deimos.io/environment-staging: deploy
finalizers:
- resources-finalizer.argocd.argoproj.io
name: node-web-app
namespace: argocd
spec:
project: klt-project
destination:
namespace: myapp
server: https://kubernetes.default.svc
source:
path: charts/generic
repoURL: https://gitlab.com/deimosdev/tooling/klt.git
targetRevision: HEAD
helm:
releaseName: node-web-app
valueFiles:
- ../../environments/overlays/local/node-web-app-values.yaml
syncPolicy:
automated:
selfHeal: true
managedNamespaceMetadata:
annotations:
keptn.sh/lifecycle-toolkit: enabled
syncOptions:
- CreateNamespace=true
retry:
limit: -1
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers:
- /spec/replicas
- group: autoscaling
kind: HorizontalPodAutoscaler
jsonPointers:
- /spec/metricsHere is the values file:
command:
- npm
- start
extraEnvVarsConfigMaps:
- node-web-app-env
fullnameOverride: node-web-app
image:
repository: localhost:5000/node-web-app
tag: 0.0.1
livenessProbe:
httpGet:
path: /healthz
port: 8081
nameOverride: node-web-app
partOf: myapp
podAnnotations:
instrumentation.opentelemetry.io/inject-nodejs: observability/intrumentation
keptn.sh/pre-deployment-tasks: pre-deployment
keptn.sh/post-deployment-tasks: post-deployment
keptn.sh/pre-deployment-evaluations: pre-deploy-eval
readinessProbe:
httpGet:
path: /healthz
port: 8081
service:
port: 8081
versionOverride: 0.1.16I just increment the versionOverride value with a new version number and then commit and push the changes, ArgoCD takes care of the rest.
The helm chart will provide the labels to the deployment as follows:
Labels: app.kubernetes.io/instance=node-web-app
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=node-web-app
app.kubernetes.io/part-of=myapp
app.kubernetes.io/version=0.1.16
helm.sh/chart=generic-0.1.9
pod-template-hash=77577587f5
Annotations: instrumentation.opentelemetry.io/inject-nodejs: observability/intrumentation
keptn.sh/post-deployment-tasks: post-deployment
keptn.sh/pre-deployment-evaluations: pre-deploy-eval
keptn.sh/pre-deployment-tasks: pre-deploymentHere are the values for the tasks and evaluations:
---
apiVersion: lifecycle.keptn.sh/v1alpha3
kind: KeptnTaskDefinition
metadata:
name: post-deployment
namespace: myapp
spec:
python:
inline:
code: |
import os
# Get environment variables
data = os.getenv('CONTEXT')
print(data)
---
apiVersion: lifecycle.keptn.sh/v1alpha3
kind: KeptnTaskDefinition
metadata:
name: pre-deployment
namespace: myapp
spec:
python:
inline:
code: |
import os
# Get environment variables
data = os.getenv('CONTEXT')
print(data)
---
apiVersion: metrics.keptn.sh/v1alpha3
kind: KeptnMetric
metadata:
name: available-cpus
namespace: klt
spec:
provider:
name: prometheus-provider
query: "sum(kube_node_status_capacity{resource='cpu'})"
fetchIntervalSeconds: 10
---
apiVersion: lifecycle.keptn.sh/v1alpha3
kind: KeptnEvaluationDefinition
metadata:
name: pre-deploy-eval
namespace: myapp
spec:
objectives:
- keptnMetricRef:
name: available-cpus
namespace: klt
evaluationTarget: ">100"
---
apiVersion: metrics.keptn.sh/v1alpha3
kind: KeptnMetricsProvider
metadata:
name: prometheus-provider
namespace: klt
spec:
type: prometheus
targetServer: http://kube-prometheus-stack-prometheus.observability.svc.cluster.local:9090Thank you, can you please also show us your KeptnEvaluation, KeptnEvaluationDefinition, KeptnMetric, KeptnMetricProvider?
pieterv-icloud-com
commented
1 year ago Sorry included in the previous comment
odubajDT
commented
1 year ago Thank you! DO you have any KeptnEvaluations in your system? Or none of them were created ?
pieterv-icloud-com
commented
1 year ago Sorry here it is:
Name: pre-eval-pre-deploy-eval-59460
Namespace: myapp
Labels: <none>
Annotations: <none>
API Version: lifecycle.keptn.sh/v1alpha3
Kind: KeptnEvaluation
Metadata:
Creation Timestamp: 2023-09-12T07:17:32Z
Generation: 1
Owner References:
API Version: lifecycle.keptn.sh/v1alpha3
Block Owner Deletion: true
Controller: true
Kind: KeptnWorkloadInstance
Name: myapp-node-web-app-0.1.16
UID: ec3bb954-d265-42f0-9e58-f30af5b2f94a
Resource Version: 12517
UID: d4b692ed-d712-4753-8eb6-98e1241412ac
Spec:
App Name: myapp
Check Type: pre-eval
Evaluation Definition: pre-deploy-eval
Retries: 10
Retry Interval: 5s
Workload: myapp-node-web-app
Workload Version: 0.1.16
Events: <none>Thank you, is this evaluation included in any of your WorkloadInstances ? From the ones you sent I cannot see it linked anywhere. It should be myapp-node-web-app-0.1.16. Do you have it in your system ?
odubajDT
commented
1 year ago I might know the reason of your problems. Previously you posted this annotation:
keptn.sh/pre-deployment-evaluations: cpu-throttling
which refers to the cpu-throttling KeptnMetric, instead, it should point to your KeptnEvaluationDefinition, so it should be the following:
keptn.sh/pre-deployment-evaluations: pre-deploy-eval
pieterv-icloud-com
commented
1 year ago I noticed yesterday that I was missing a KeptnEvaluationDefinition in my configuration.
So I did annotate the workload with:
keptn.sh/pre-deployment-evaluations: pre-deploy-eval
odubajDT
commented
1 year ago OK and you still get the same error ?
pieterv-icloud-com
commented
1 year ago Think I found the issue, the KeptnEvaluationDefinition was not right:
apiVersion: lifecycle.keptn.sh/v1alpha3
kind: KeptnEvaluationDefinition
metadata:
name: pre-deploy-eval
namespace: myapp
spec:
objectives:
- keptnMetricRef:
name: available-cpus
namespace: klt
evaluationTarget: ">100"It should have been:
apiVersion: lifecycle.keptn.sh/v1alpha3
kind: KeptnEvaluationDefinition
metadata:
name: pre-deploy-eval
namespace: myapp
spec:
objectives:
- keptnMetricRef:
name: available-cpus
namespace: klt
evaluationTarget: ">100"Another issue was the metric evaluation was wrong as well, under my minikube configuration the amount of cpus available is only 4:
NAMESPACE NAME PROVIDER QUERY INTERVAL VALUE
klt available-cpus prometheus-provider sum(kube_node_status_capacity{resource='cpu'}) 4I had to dial the evaluation down to something like this:
apiVersion: lifecycle.keptn.sh/v1alpha3
kind: KeptnEvaluationDefinition
metadata:
name: pre-deploy-eval
namespace: myapp
spec:
objectives:
- keptnMetricRef:
name: available-cpus
namespace: klt
evaluationTarget: ">1"Looks like it's working now.
odubajDT
commented
1 year ago Awesome, thank you for the update. If the issue is resolved now, I will close this ticket. If you find any other issue, feel free to reopen it
pieterv-icloud-com
commented
1 year ago Thx really appreciate the assistance
Using the 0.8.2 release currently.
When increment the version of my application from 0.1.5 to 0.1.6 I can see the following int he KeptnWorkloadInstance:
The warning here:
Seems like it is looking for an incorrect app should this not be
myappinstead ofmyapp-node-web-app-0.1.6?My workload remains in a Pending state