[Feature] Create ephemeral namespace in all registered clusters

mikebryant commented 3 weeks ago

Problem Statement

We have a multi-cloud multi-cluster environment. We want to do a multi-cloud istio test

Current approach is something like: chainsaw test --cluster aws=${HOME}/.kube/boxed-config.yaml:sbx-aws --cluster gcp=${HOME}/.kube/boxed-config.yaml:sbx-gcp --selector chainsaw.nwboxed.com/multi-cloud=enabled chainsaw

Our test looks like this:

---
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
  name: istio-multi-cloud-communication
  labels:
    chainsaw.nwboxed.com/multi-cloud: enabled
spec:
  cluster: aws
  steps:
  - try:
    - apply:
        cluster: aws
        file: create-deployments-client.yaml
    - apply:
        cluster: gcp
        file: create-deployments-server.yaml

But we get the following:

Running tests...
=== RUN   chainsaw
=== PAUSE chainsaw
=== CONT  chainsaw
=== RUN   chainsaw/istio-multi-cloud-communication
=== PAUSE chainsaw/istio-multi-cloud-communication
=== CONT  chainsaw/istio-multi-cloud-communication
    | 12:43:31 | istio-multi-cloud-communication | @setup   | CREATE    | OK    | v1/Namespace @ chainsaw-positive-moray
    | 12:43:31 | istio-multi-cloud-communication | step-1   | TRY       | RUN   |
    | 12:43:31 | istio-multi-cloud-communication | step-1   | APPLY     | RUN   | apps/v1/Deployment @ chainsaw-positive-moray/sleep
    | 12:43:31 | istio-multi-cloud-communication | step-1   | CREATE    | OK    | apps/v1/Deployment @ chainsaw-positive-moray/sleep
    | 12:43:31 | istio-multi-cloud-communication | step-1   | APPLY     | DONE  | apps/v1/Deployment @ chainsaw-positive-moray/sleep
    | 12:43:31 | istio-multi-cloud-communication | step-1   | APPLY     | RUN   | v1/Service @ chainsaw-positive-moray/helloworld
    | 12:43:32 | istio-multi-cloud-communication | step-1   | CREATE    | ERROR | v1/Service @ chainsaw-positive-moray/helloworld
        === ERROR
        namespaces "chainsaw-positive-moray" not found

It's creating the ephemeral Namespace only on one cluster, not both

Solution Description

For the ephemeral Namespace to be created on all clusters

Alternatives

No response

Additional Context

No response

Slack discussion

No response

Research

[X] I have searched other issues in this repository and mine is not recorded.

mikebryant commented 3 weeks ago

I've worked around this for the moment with the following:

spec:
  cluster: aws
  steps:
  - try:
    - script:
        cluster: aws
        env:
        - name: NAMESPACE
          value: ($namespace)
        content: |
          #!/bin/bash
          set -e
          kubectl get namespace ${NAMESPACE} -o json | jq 'del(.metadata.resourceVersion, .metadata.uid, .metadata.annotations, .metadata.creationTimestamp, .metadata.selfLink, .metadata.managedFields, .status, .spec.finalizers)' > ./tmp-namespace.json
    cleanup:
    - script:
        cluster: aws
        content: |
          #!/bin/bash
          set -e
          rm ./tmp-namespace.json
  - try:
    - apply:
        cluster: gcp
        file: ./tmp-namespace.json

eddycharly commented 3 weeks ago

Thanks @mikebryant ! I'm busy with v1alpha2 test schema right now but i can prepare the ground for this.

I feel like users will want this while others won't, this makes sense to make it configurable.

kyverno / chainsaw