Open diranged opened 1 week ago
ashishdhingra
commented
1 week ago @diranged Good morning. Could you please confirm if this is a CDK issue and share minimal code to reproduce the issue? Or this issue was originally intended for https://github.com/kubernetes/kubectl/ repo?
Thanks, Ashish
diranged
commented
1 week ago Honestly I think this is a CDK issue because the overwrite: true flag is no longer behaving the way the user expects...
pahud
commented
1 week ago Hi
@diranged
Can you share a minimal CDK app that we can test and reproduce this issue in our account?
And please let us know which version was working and which version is broken now with exactly the same code.
Thanks.
diranged
commented
1 week ago @pahud, I can try - but I don't know when I'll have time to get that done... but I will note that rolling back to the V27 Lambda Kubectl function resolves the issue.
pahud
commented
1 week ago The error message "appprojects.argoproj.io "default" already exists" indicates that you're trying to create an Argo CD AppProject resource named "default" in the "argocd-system" namespace, but a resource with the same name already exists in your cluster. [1]
This situation typically occurs when:
You've previously created an AppProject named "default" in the same namespace.
The "default" AppProject was automatically created during the Argo CD installation process.
Argo CD typically creates a "default" AppProject during its initial setup, which is why you're encountering this error when trying to apply your manifest.
To resolve this issue:
Update instead of create: If you want to modify the existing "default" AppProject, you can use kubectl apply with the --force flag:
kubectl apply -f your-manifest.yaml --forcelooks like there's already a default AppProject and you are install another one with the same name? I am not sure if this is related to CDK but the issue https://github.com/kubernetes/kubernetes/issues/44165 you mentioned is in 2017 and I am not sure if this is related to CDK.
I am not the expert of ArgoCD but hope this help.
diranged
commented
1 week ago @pahud,
So this code has been in place and untouched (other than updates to aws-cdk-lib, aws-cdk and the @aws-cdk/lambda-layer-kubectl-v28 typescript libraries) for 2 years now. It started breaking after some recent update (though it's hard right now for me to pinpoint it, because we don't run integration tests 100% of the time). Yes, ArgoCD auto-creates the default AppProject object on startup - it's a foot-gun being discussed at https://github.com/argoproj/argo-cd/issues/11058 ... however, this behavior has been in place for several years.
Given the following code:
const cdk8sPostChart = new cdk8s.Chart(new cdk8s.App(), 'PostManifestBuilder', {
namespace: this.namespace,
});
new AppProject(cdk8sPostChart, 'DefaultProject', {
metadata: { name: 'default' },
spec: props.defaultProjectSpec ?? DEFAULT_PROJECT_SPEC,
});
new KubernetesManifest(this, 'PostHelmResources', {
cluster: this.cluster,
overwrite: true, // the argo controller creates a default 'appProject' on startup, we overwrite it
prune: true,
manifest: cdk8sPostChart.toJson(),
}).node.addDependency(helmChart);One would expect that regardless of whether the default object already exists or not, it would be overwritten via kubectl apply -f manifest.yaml... but it seems that in some cases that does not happen. I've tried to replicate this with a local kind environment using pure kubectl commands and for some reason I cannot .... which leads me to believe that there's actually a race-condition happening that is made worse by the Server Side Apply setup. The first create call is at 2024-10-05T18:12:49.590120Z and the second one comes in a hair later at 2024-10-05T18:12:49.605439Z.
Do you have specific objections to either:
a) exposing --server-side=<bool> an option for the KubernetesManifest resource
b) using --server-side=false when overwrite==true
c) using --force-conflicts=true when overwrite==true
Describe the bug
At some point recently, the
kubectlCLI started setting--server-side-applyas the default behavior. The problem is that with https://github.com/kubernetes/kubernetes/issues/44165,kubectl apply -f ...no longer works the way you'd expect. On a server-side-apply, it seems that Kubernetes will refuse to update a resource that already exists, which then reports back to the Lambda function anAlreadyExistserror.Regression Issue
Last Known Working CDK Version
unknown
Expected Behavior
I would expect that
kubectl apply -f ...just works ... (which is configured by settingoverwrite: trueon theKubernetesManifestresource)... but instead it's failing.Current Behavior
Here are the logs from the Lambda function trying to run
kubectl apply -fon a resource that happens to already exist in the cluster:If we go and look at the Kubernetes Audit logs, we can see that ArgoCD first creates this
defaultresource, and then the nextCreatecall fails with a409and is a server-side-apply call:Here's the first
createcall (made by ArgoCD, and uncontrollable by us)Then we see a second call, this time via
kubectl... note thekubectl-client-side-applyin therequestURIpath:Reproduction Steps
N/A
Possible Solution
I think that when
overwrite: trueis set, then the--server-side=falseflag should also be applied to the command..Additional Information/Context
No response
CDK CLI Version
2.161.1
Framework Version
No response
Node.js Version
18
OS
linux
Language
TypeScript
Language Version
No response
Other information
No response