dag-andersen / argocd-diff-preview

Tool for rendering manifest changes on pull requests.
Apache License 2.0
156 stars 10 forks source link

Support for ArgoCD config management plugins #40

Closed davidfrickert closed 2 weeks ago

davidfrickert commented 2 months ago

Great project idea, I see in the README that CMP are not supported at the moment though.

Is this planned to be implemented, or not possible at all to implement?

Would be great to have support previewing changes of applications using argocd-lovely and other plugins.

davidfrickert commented 2 months ago

Ok, this can be achieved by using kustomize

PoC available here: https://github.com/davidfrickert/argocd-diff-preview/

dag-andersen commented 2 months ago

Hi @davidfrickert - Thank you for showing interest in the project :)

not supported

What this means is that CMPs haven’t been tested yet, so I can’t confirm if they work properly with the tool.

I haven’t had the chance to explore CMPs much myself, so my experience with them is somewhat limited. From what I understand, there are a variety of ways to install and configure CMPs, which makes it tricky to develop a solution that supports every possible case while still keeping the tool user-friendly and customizable.

That said, if you have specific suggestions or ideas on how to integrate CMPs, I’d be happy to discuss them further!

And thank you for showing me your PoC. Maybe I should support adding kustomize patches to the argocd installation ⬇️ In this case you could put your patch file in a kustomize-patches folder and it would be applied when installing argocd ⬇️ How does that sound?

jobs:
  build:
    ...
    steps:
      ...
    - name: Generate Diff
      run: |
        docker run \
          --network=host \
          -v /var/run/docker.sock:/var/run/docker.sock \
          -v $(pwd)/main:/base-branch \
          -v $(pwd)/pull-request:/target-branch \
          -v $(pwd)/output:/output \
          -v $(pwd)/kustomize-patches:/kustomize-patches \          ⬅️ Place all patches here
          -e TARGET_BRANCH=${{ github.head_ref }} \
          -e REPO=${{ github.repository }} \
          dagandersen/argocd-diff-preview:v0.0.15
davidfrickert commented 2 months ago

Hi @davidfrickert - Thank you for showing interest in the project :)

not supported

What this means is that CMPs haven’t been tested yet, so I can’t confirm if they work properly with the tool.

I haven’t had the chance to explore CMPs much myself, so my experience with them is somewhat limited. From what I understand, there are a variety of ways to install and configure CMPs, which makes it tricky to develop a solution that supports every possible case while still keeping the tool user-friendly and customizable.

That said, if you have specific suggestions or ideas on how to integrate CMPs, I’d be happy to discuss them further!

And thank you for showing me your PoC. Maybe I should support adding kustomize patches to the argocd installation ⬇️ In this case you could put your patch file in a kustomize-patches folder and it would be applied when installing argocd ⬇️ How does that sound?


jobs:
  build:
    ...
    steps:
      ...
    - name: Generate Diff
      run: |
        docker run \
          --network=host \
          -v /var/run/docker.sock:/var/run/docker.sock \
          -v $(pwd)/main:/base-branch \
          -v $(pwd)/pull-request:/target-branch \
          -v $(pwd)/output:/output \
          -v $(pwd)/kustomize-patches:/kustomize-patches \          ⬅️ Place all patches here
          -e TARGET_BRANCH=${{ github.head_ref }} \
          -e REPO=${{ github.repository }} \
          dagandersen/argocd-diff-preview:v0.0.15
No problem!

Yeah this sounds good. From my understanding CMPs are always atleast a container in the Repo server pod, so kustomize patches passed to the container should be enough!

dag-andersen commented 2 months ago

Cool - i'll write that on my todo-list! :) Thank you for this feature suggestion!

dag-andersen commented 2 months ago

Hi @davidfrickert,

I’ve just released version 0.0.18, which installs Argo CD via a Helm Chart. This allows users to customize any Argo CD parameter they wish (as long as it’s supported by the Helm chart, of course).

From a tool perspective, supporting custom Argo CD installations (and CMPs) with a Helm chart is easier compared to Kustomize, since users only need to provide a single values.yaml file with their configuration overrides.

I believe you can install argocd-lovely through the Helm Chart. The chart includes values like repoServer.extraContainers and repoServer.volumes, which allow you to add your argocd-lovely-plugin container and volume. If you encounter any issues, please let me know.

Info on how to override the Argo CD installation

davidfrickert commented 2 months ago

that should work fine! i'll report back when i have tested it

dag-andersen commented 2 weeks ago

@davidfrickert Did you have time to test it? 🚀

davidfrickert commented 2 weeks ago

I wanted to but then I remembered I had another patch on my fork which is to be able to parametrize the kind cluster name due to using this in CI 😅

dag-andersen commented 2 weeks ago

We can just add an option to specify "cluster-name" @davidfrickert! Just like you did in your PoC You can create the PR in case you want a contribution on the repo :) otherwise, I'll just do it in the next release 🚀

davidfrickert commented 2 weeks ago

We can just add an option to specify "cluster-name" @davidfrickert! Just like you did in your PoC You can create the PR in case you want a contribution on the repo :) otherwise, I'll just do it in the next release 🚀

Great, I will try to submit a PR this week!

dag-andersen commented 2 weeks ago

@davidfrickert Since the cluster name is now customizable as part of v0.0.24, I look forward to hearing if you can get it to work with your plugin! 🚀

davidfrickert commented 2 weeks ago

went ahead and tested it but doesn't seem to work @dag-andersen

for some reason, the tool is ignoring many of my applications:

✨ Running with:
✨ - local-cluster-tool: Kind
✨ - base-branch: master
✨ - target-branch: fix/use-official-argocd-diff-preview-image
✨ - secrets-folder: ./secrets
✨ - output-folder: ./output
✨ - repo: davidfrickert/gitops-apps
✨ - timeout: 120 seconds
✨ - file-regex: rendered-apps.yaml
🤖 Fetching all files in dir: base-branch
🤖 Found 42 applications
🤖 Patching applications for branch: master
🤖 Patching 42 Argo CD Application[Sets] for branch: master
🤖 Fetching all files in dir: target-branch
🤖 Found 42 applications
🤖 Patching applications for branch: fix/use-official-argocd-diff-preview-image
🤖 Patching 42 Argo CD Application[Sets] for branch: fix/use-official-argocd-diff-preview-image
🚀 Creating cluster...
🚀 Cluster created successfully
🤫 Applied 1 secrets
🦑 Installing Argo CD Helm Chart version: 'latest'
🦑 Waiting for Argo CD to start...
🦑 Argo CD is now available
🦑 Logging in to Argo CD through CLI...
🦑 Argo CD installed successfully
🌚 Getting resources from base-branch
⏳ Waiting for 15 out of 15 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 120 seconds...
⏳ Waiting for 15 out of 15 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 115 seconds...
⏳ Waiting for 15 out of 15 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 110 seconds...
⏳ Waiting for 15 out of 15 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 105 seconds...
⏳ Waiting for 2 out of 15 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 99 seconds...
🌚 Got all resources from 15 applications for base
🧼 Removing applications
🧼 Removed applications successfully
🌚 Getting resources from target-branch
⏳ Waiting for 15 out of 15 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 120 seconds...
⏳ Waiting for 1 out of 15 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 114 seconds...
🌚 Got all resources from 15 applications for target
💥 Deleting cluster...
🔮 Generating diff between master and fix/use-official-argocd-diff-preview-image
🙏 Please check the ./output/diff.md file for differences
🎉 Done in 140 seconds
Command completed after 1 attempt(s).

this doesn't happen on my patched image:

✨ Running with:
✨ - local-cluster-tool: Kind
✨ - base-branch: master
✨ - target-branch: renovate/velero-velero-plugin-for-aws-1.x
✨ - secrets-folder: ./secrets
✨ - output-folder: ./output
✨ - repo: davidfrickert/gitops-apps
✨ - timeout: 120 seconds
✨ - file-regex: rendered-apps.yaml
🚀 Creating cluster...
🚀 Cluster created successfully
🦑 Installing Argo CD version: 'stable'
🦑 Waiting for Argo CD to start...
🦑 Argo CD is now available
🦑 Logging in to Argo CD through CLI...
🦑 Argo CD installed successfully
🤫 Applied 1 secrets
🤖 Fetching all files in dir: base-branch
🤖 Patching applications for branch: master
🤖 Patching 42 Argo CD Application[Sets] for branch: master
🤖 Fetching all files in dir: target-branch
🤖 Patching applications for branch: renovate/velero-velero-plugin-for-aws-1.x
🤖 Patching 42 Argo CD Application[Sets] for branch: renovate/velero-velero-plugin-for-aws-1.x
🌚 Getting resources for base
⏳ Waiting for 42 out of 42 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 120 seconds...
⏳ Waiting for 18 out of 42 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 113 seconds...
⏳ Waiting for 17 out of 42 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 107 seconds...
⏳ Waiting for 15 out of 42 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 102 seconds...
⏳ Waiting for 15 out of 42 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 97 seconds...
⏳ Waiting for 2 out of 42 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 91 seconds...
🌚 Got all resources from 42 applications for base
🧼 Removing applications
🧼 Removed applications successfully
🌚 Getting resources for target
⏳ Waiting for 15 out of 42 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 118 seconds...
⏳ Waiting for 4 out of 42 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 111 seconds...
⏳ Waiting for 1 out of 42 applications to become 'OutOfSync'. Retrying in 5 seconds. Timeout in 106 seconds...
🌚 Got all resources from 42 applications for target
💥 Deleting cluster...
🔮 Generating diff between master and renovate/velero-velero-plugin-for-aws-1.x
🙏 Please check the ./output/diff.md file for differences
🎉 Done in 168 seconds
Command completed after 1 attempt(s).
davidfrickert commented 2 weeks ago

So I just added the custom values and mounted the file to the container and changed to v0.0.24 image from my patched image

      - name: Set ArgoCD Custom Values
        run: |
          cat > values.yaml << "EOF"
          repoServer:
            extraContainers:
            # ArgoCD Lovely plugin - https://github.com/crumbhole/argocd-lovely-plugin
              - name: lovely-plugin
                image: ghcr.io/crumbhole/lovely:1.1.1
                securityContext:  
                  runAsNonRoot: true
                  runAsUser: 999
                volumeMounts:
                    # Import the repo-server's plugin binary
                  - mountPath: /var/run/argocd
                    name: var-files
                  - mountPath: /home/argocd/cmp-server/plugins
                    name: plugins
                  - mountPath: /tmp
                    name: lovely-tmp
            volumes:
              - emptyDir: {}
                name: lovely-tmp
          EOF

      - name: Generate Diff
        id: argocd-diff-preview
        uses: nick-fields/retry@v3
        with:
          timeout_minutes: 6
          max_attempts: 5
          continue_on_error: true
          command: |
            docker run \
              --network=host \
              -v $(pwd)/master:/base-branch \
              -v $(pwd)/pull-request:/target-branch \
              -v $(pwd)/output:/output \
              -v $(pwd)/secrets:/secrets \
              -v $(pwd)/values.yaml:/argocd-config/values.yaml \
              -e BASE_BRANCH=${{ github.event.pull_request.base.ref }} \
              -e TARGET_BRANCH=${{ github.head_ref }} \
              -e REPO=${{ github.repository }} \
              -e TIMEOUT=120 \
              -e FILE_REGEX=rendered-apps.yaml \
              -e CLUSTER_NAME=argocd-diff-preview-${{steps.kind-id.outputs.id}} \
              -e DOCKER_HOST="tcp://localhost:2376" \
              -e DOCKER_TLS_CERTDIR=/certs \
              -e DOCKER_CERT_PATH=/certs/client \
              -e DOCKER_TLS_VERIFY=1 \
              -e DOCKER_BUILDKIT=1 \
              -v /certs:/certs \
              dagandersen/argocd-diff-preview:v0.0.24
davidfrickert commented 2 weeks ago

For some reason the number 15 exactly matches the number of applications I have that refer to a git repo location (with or without CMP plugin), so all the "native" helm applications are being ignored. interesting

dag-andersen commented 2 weeks ago

@davidfrickert - Since v0.0.22, the tool ignores Applications whose files haven't changed and that don't contain any references to the repository it’s running on. This is because rendering these apps would produce the same result for both branches, so there's no need to process them. I believe the behavior you're seeing is expected.

You can run the tool with --debug to see more details about what's happening. It should log messages like Skipping application XXX, because it has not changed for each skipped app.

But that's a good point! I need to add a few log messages describing how/why some of the apps are being ignored. If it is not running in debug, it seems like it randomly goes from 42 to 15 apps.

davidfrickert commented 2 weeks ago

@davidfrickert - Since v0.0.22, the tool ignores Applications whose files haven't changed and that don't contain any references to the repository it’s running on. This is because rendering these apps would produce the same result for both branches, so there's no need to process them. I believe the behavior you're seeing is expected.

You can run the tool with --debug to see more details about what's happening. It should log messages like Skipping application XXX, because it has not changed for each skipped app.

But that's a good point! I need to add a few log messages describing how/why some of the apps are being ignored. If it is not running in debug, it seems like it randomly goes from 42 to 15 apps.

Aha okay, that explains it. Then all good I guess :)

davidfrickert commented 2 weeks ago

Thanks for the great work @dag-andersen !

dag-andersen commented 2 weeks ago

@davidfrickert Great! So, you got it working with the code you shared in this comment, and it worked as expected? If so, I'll add that to the documentation as an example 🚀

davidfrickert commented 2 weeks ago

@davidfrickert Great! So, you got it working with the code you shared in this comment, and it worked as expected? If so, I'll add that to the documentation as an example 🚀

Yep I think so, although that is only a snippet, would you like the full workflow?

Anyways, I will be monitoring along the next days how this handles incoming updates in my repo!

dag-andersen commented 2 weeks ago

@davidfrickert No, thank you :) The snippet is fine, I only need the "Set ArgoCD Custom Values"-step. I updated the docs: Argo CD Config Management Plugins (CMP)