[Feature] ArgoCD Integration

[1ddo]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Use case

Most of our CD are done with Kubernetes ArgoCD and we wanted to add this data connection to DevLake so we could leverage DORA for our deployment frequency.

Description

Add Kubernetes ArgoCD as new data connection to DevLake

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[mintsweet]

hi @1ddo, thank you for your report.

We have received your information and will consider your needs.

[klesh]

@1ddo Hi, I see you checked the Yes I am willing to submit a PR!, would you like to work on it? If so, we could assign the issue to you and you can start working on it.

[1ddo]

Hi @klesh , I'm currently working on it locally, I'm new to this stuff and I'm having difficulty in making devlake work locally in windows. Was able to run the components in Docker Windows however I'm getting API Down in the config UI (localhost:4000), the tutorial said just wait for a few minutes to let the API up and running but mines seems not working. Do you have any more docs on setting devlake locally using windows or any docs that could help me speed things up?

[klesh]

@1ddo Hi, I'm using Windows too because of Feishu(Good collaborative software but lousy Linux support). Sadly, development in Native Windows is not supported, I am using WSL2, which is practically an Ubuntu Linux environment in conjunction with VSCode Remote WSL plugin, you may take a look, it is a far superior development experience than Native Windows.

[1ddo]

Hi @klesh, thanks for the advice. Will try to set up a WSL2

[1ddo]

Hi @klesh , do you know the reason why I don't have a Project menu in my config-ui:4000? I only see Data Connections, Blue Print, Disabled button for Connection and Dashboard menu options?

[abeizn]

@1ddo It should be that your version is not the latest, you can use v0.17.0-beta5 version, the installation method can refer to here: https://devlake.apache.org/docs/next/GettingStarted/DockerComposeSetup

[klesh]

Hi @klesh , do you know the reason why I don't have a Project menu in my config-ui:4000? I only see Data Connections, Blue Print, Disabled button for Connection and Dashboard menu options?

How did you launch devlake backend and config-ui ? Were you using the docker version for the config-ui?

[1ddo]

I've been working all day with the upgrade, by default the version value for the config-ui, grafana and devlake is set to "latest" then I replaced it with the version "v0.17.0-beta5" then ran docker-compose up -d but after that I encountered lots of issues one after another like token, certificate issue, installing grafana cli and others. Till now I'm working with those issues. Did you encounter this issue before? I'm using WSL Ubuntu by the way.

Here's one of the issue I'm getting:

failed to solve: failed to fetch anonymous token: Get "https://auth.docker.io/token?scope=repository%3Alibrary%2Fpython%3Apull&service=registry.docker.io": tls: failed to verify certificate: x509: certificate signed by unknown authority

For the Docker, I've connected my Docker Desktop to WSL Ubuntu and I have the latest Docker Desktop version and was able to run the 4 components (devlake, mysql, config-ui, grafana) but since I'm currently upgrading it to the latest version I haven't tried to run them again since I have a bunch of issues to work on.

[BrookeKatalon]

I'm also interested in the ArgoCD connection

[1ddo]

Hi @BrookeKatalon, are you willing to contribute with the ArgoCD development? If yes then we could collab with it.

[klesh]

@1ddo I'm confused, are you trying to code for devlake or just use it to collect and observe data? For the former, you don't need the docker-compose for bringing up devlake and config-ui, please follow this guide to setup the Development Env https://devlake.apache.org/docs/next/DeveloperManuals/DeveloperSetup For the latter, don't use the docker-compose.yml from the repo, download the one from the release page.

The error message seems like a network problem

[1ddo]

Hi @klesh , we need both since we need collect Data and observed data from ArgoCD but since it's not yet supported then we need to integrate it with DevLake. So don't we need to make DevLake work locally first before start coding, that way we could verify/test it later?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[klemen-df]

What's the status on this?

[klesh]

@KlemenDanfoss Under discussion, no plan or volunteer at this point AFAIK 😂

[gespi1]

a requirement for the pluginshould be to support argocd multisource applications. Currently devlake takes in one value for repo_url. For argocd apps with mulitple sources the repourl returns a list of multiple repoURL's

[pratiyush05]

Hi @klesh ,we have a scenario with ArgoCD too. We have multiple repos for a single project and then a single team repo . Team repo has image version and from here ArgoCD takes up image versions and do its deployment. So only this team repo's main HEAD commit_sha gets noted in Devlake production deployment from ArgoCD webhook .

We would like to know how Devlake can club multiple repo deployments into a single deployment . In our case , there shall be multiple github repo in a Devlake project and a team repo present in multiple Devlake projects. Plus ArgoCD linked to a single team repo's main branch's HEAD commit only . Could you please suggest a project configuration for us ?

Like clubbing of multiple deployments into one as shown here in blue one .

[klesh]

@pratiyush05 you may create deployment via the webhook plugin , but it actually are creating cicd_deployment_commit records at the moment. We are planning to support multiple repos in the near future by https://github.com/apache/incubator-devlake/issues/6262

[pratiyush05]

Hi @klesh

Above diagram shows our current setup . So we would need to add github data connection in this way format - Devlake_Proj_1 - Github Repo_1 + Repo_7 Devlake_Proj_2 - Github Repo_2 + Repo_7 Devlake_Proj_3 - Github Repo_3 + Repo_7 Devlake_Proj_4 - Github Repo_4 + Repo_7 ... total 6 Devlake_projs

My query is how to get deployments of Repo_1 to Repo_6 in their Devlake projects , so that the graph "Median Lead Time to Change" correctly get deployments related to a project . ArgoCD can send commits of Repo_7 only . Since that graph first finds deployments in repos , then the commits and PRs of those commits . So we would need commits from Repo_1 to Repo_6 in Devlake as well .

We are not using Github actions for deployment , so we cannot track deployment in github plugin itself .

[nirmaljeet-singh-groww]

Any idea when can this be releasesd for integration with argocd?

[thiDucTran]

hi, i am willing to help in getting argocd supported. currently, we have a mixed setup. for scm we uses azure repo. for CD, we uses both ArgoCD and Azure DevOps .. with argocd, planning to do https://argocd-notifications.readthedocs.io/en/stable/services/webhook/#send-form-data ..is this the right way to go about it?

[d4x1]

@thiDucTran I think we should add an ArgoCD plugin and collect deployments(and other data) from ArgoCD, then we can combine data from AzureDevops and ArgoCD into one dashboard with some SQL.

[thiDucTran]

i think the issue, for us, is that our ArgoCD auto sync and deploys based on non-application repos (basically whenever argocd see changes to our kustomize overlays)... meaning there needs to be a way to correlate the fact that an ArgoCD deployment was a deployment of which app, which commit of that app etc..

[d4x1]

Can Applications in ArgoCD be related to Azure Repos ?

[YuseopKim]

In my opinion, to use ArgoCD, DevLake needs to make a relationship between deployment repository commit (for manifests) and application repository commit (for source codes).

[klesh]

@YuseopKim We need a separate repo to host ArgoCD stuff, right? We can create one if someone is interested in working on it.

[thiDucTran]

In my opinion, to use ArgoCD, DevLake needs to make a relationship between deployment repository commit (for manifests) and application repository commit (for source codes).

im thinking about the same. one way that i can think of is this flow:

1. CI tool (jenkins, tekton, any CI tool) builds from application repo and produces new docker image. the same CI tool would create a PR to update deployment repo (update manifests to use new docker image); when creating the PR ...we can put the commit hash of the app's repo some where..part of the docker image's tag ..part of the PR's name? this is because when ArgoCD deploys the new docker image ..upon deployment completion, ArgoCD can do some parsing to get the app repo's commit hash and POST it to devlake via webhook?

[thiDucTran]

---edit--- For DORA, I found the payload schemas related to incidents and deployments. Regarding ArgoCD, I am only interested in deployments. Seems like what I need is https://devlake.apache.org/docs/Plugins/webhook/#deployment . FYI, incident schemas are https://devlake.apache.org/docs/Plugins/webhook/#register-issues---close-issues-optional and https://devlake.apache.org/docs/Plugins/webhook/#register-issues---update-or-create-issues

---

hi team, while I understand that the API References are available.....can someone provide a list of RESTful APIs related to DORA? specifically anything that is related to deployments (relating to ArgoCD)...this is an attempt to do a proof of concept as described in https://github.com/apache/incubator-devlake/issues/5207#issuecomment-1958832175

[thiDucTran]

edit: already resolved in https://devlake-io.slack.com/archives/C03APJ20VM4/p1712898142223339

referencing https://devlake.apache.org/docs/Plugins/webhook/#deployment . I am making the REST call as below but is seeing the error: {"success":false,"message":"path doesn't match api key's scope"} ...what path is the error speaking of?

curl 'http://apache-devlake-ui.infra.svc:4000/api/rest/plugins/webhook/1/deployments' -X 'POST' -H 'Authorization: Bearer abc123' -d '{
    "repo_url":"https://github.com/apache/incubator-devlake/",
    "commit_sha":"015e3d3b480e417aede5a1293bd61de9b0fd051d",
    "commit_msg":"optional-commit-message",
    "start_time":"2020-01-01T12:00:00+00:00",
    "end_time":"2020-01-02T13:00:00+00:00",
    "result": "FAILURE"
  }' 

[d4x1]

/api/rest/plugins/webhook/1/deployment

Check your API key abc123 's allowed path config. I think the regexp you configed cannot match /api/rest/plugins/webhook/1/deployments.

[thiDucTran]

this is what I have so far..i have not tested all cases like for argocd sync failtures but I know it works when argocd syncs the app changes successfully....there is one thing about commit_sha and how we can pass this info to ArgoCD..can we use some sort of annotation from the ArgoCD Application itself? will continue to test

apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
  namespace: default
stringData:
  devlake-basic-auth: 'foobar'
type: Opaque
---
# $devlake-basic-auth is from argocd-notifications-secret
apiVersion: v1
data:
  context: |
    argocdUrl:
  service.webhook.devlake: |
    url: http://apache-devlake-ui.infra.svc:4000
    headers:
    - name: Authorization
      value: Basic $devlake-basic-auth

  template.devlake-commit-status: |
    webhook:
      devlake:
        method: POST
        path: /api/plugins/webhook/connections/1/deployments
        body: |
          {
            {{if eq .app.status.operationState.phase "Running"}} "result": "FAILURE"{{end}}
            {{if eq .app.status.operationState.phase "Succeeded"}} "result": "SUCCESS"{{end}}
            {{if eq .app.status.operationState.phase "Error"}} "result": "FAILURE"{{end}}
            {{if eq .app.status.operationState.phase "Failed"}} "result": "FAILURE"{{end}},
            "repo_url": "https://yourURLhere",
            "commit_sha": "69ba6b8755848540c88577e4a5e23e14a5d63958",
            "commit_msg": "optional-commit-message",
            "start_time": "{{.app.status.operationState.startedAt}}",
            "end_time": "{{.app.status.operationState.finishedAt}}",
            "environment": "PRODUCTION"
          }
  trigger.on-deployed: |
    - when: app.status.operationState.phase in ['Succeeded']
      send: [devlake-commit-status]
  trigger.sync-operation-change: |
    - when: app.status.operationState.phase in ['Error', 'Failed']
      send: [devlake-commit-status]
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: notifications-controller
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-notifications-controller
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: v2.9.9
    argocd.argoproj.io/instance: argocd
    helm.sh/chart: argo-cd-5.54.0
  name: argocd-notifications-cm
  namespace: default
---
# notice the annotations to subscribe to the webhook notification to whichever argocd app you want
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-deployed.devlake: ""
    notifications.argoproj.io/subscribe.sync-operation-change.devlake: ""                                                                                                                                                                                                                                     name: thi-dora                                                                                                                                                                                                   namespace: default

[thiDucTran]

this is what I ended up with..I hope someone else may be can come up with a better way... for hash or commit sha of your app.. it is just a matter of having the annotation in your defined argocd app....also don't forget to also add more annotations to subscribe to your defined triggers.... as for devlake integration or having devlake doing all this... I'm not sure how you would go about it

  template.devlake-commit-status: |
    webhook:
      devlake:
        method: POST
        path: /api/plugins/webhook/connections/1/deployments
        body: |
          {
            {{if eq .app.status.health.status "Healthy"}} "result": "SUCCESS"{{end}}
            {{if ne .app.status.health.status "Healthy"}} "result": "FAILURE"{{end}},
            "repo_url": "https://yourUrlHere",
            "commit_sha": "{{.app.metadata.annotations.hash}}",
            "commit_msg": "optional-commit-message",
            "start_time": "{{.app.status.operationState.startedAt}}",
            "end_time": "{{.app.status.operationState.finishedAt}}",
            "environment": "PRODUCTION"
          }
  trigger.on-deployed: |
    - when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status != ''
      send: [devlake-commit-status]
  trigger.on-health-degraded: |
    - when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status != ''
      send: [devlake-commit-status]
  trigger.on-sync-succeeded: |
    - when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status != ''
      send: [devlake-commit-status]

[d4x1]

@thiDucTran Nice work!

I am not familiary with ArgoCD, I can't do anything helpful. What you have done is very helpful and I think it will work.

[KelvinVenancio]

thanks, @thiDucTran, I was searching for exactly that. I'll give back here soon with the feedback.

[mars-esther]

Is there anyway to give repo url dynamically also? because we are using almost 300 repos and im not sure how to configure it

[dosubot[bot]]

Hey @mars-esther! I'm a bot designed to help you out with bugs, questions, and becoming a contributor. How can I assist you today?

Yes, you can dynamically manage multiple repositories in Apache DevLake using the PutScopes API. This API allows you to create or update multiple repositories in bulk for tools like Azure DevOps and Bitbucket.

Here is an example for Azure DevOps:

// PutScopes create or update Azure DevOps repo
// @Summary create or update Azure DevOps repo
// @Description Create or update Azure DevOps repo
// @Tags plugins/azuredevops
// @Accept application/json
// @Param connectionId path int true "connection ID"
// @Param scope body PutScopesReqBody true "json"
// @Success 200  {object} []models.AzuredevopsRepo
// @Failure 400  {object} shared.ApiBody "Bad Request"
// @Failure 500  {object} shared.ApiBody "Internal Error"
// @Router /plugins/azuredevops/connections/{connectionId}/scopes [PUT]
func PutScopes(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
    return dsHelper.ScopeApi.PutMultiple(input)
}

And for Bitbucket:

// PutScopes create or update repo
// @Summary create or update repo
// @Description Create or update repo
// @Tags plugins/bitbucket
// @Accept application/json
// @Param connectionId path int true "connection ID"
// @Param scope body PutScopesReqBody true "json"
// @Success 200  {object} []models.BitbucketRepo
// @Failure 400  {object} shared.ApiBody "Bad Request"
// @Failure 500  {object} shared.ApiBody "Internal Error"
// @Router /plugins/bitbucket/connections/{connectionId}/scopes [PUT]
func PutScopes(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
    return dsHelper.ScopeApi.PutMultiple(input)
}

These endpoints allow you to send a PUT request with the necessary data to create or update multiple repositories at once ^[1][2].

To continue talking to Dosu, mention @dosu.

[klesh]

Is there anyway to give repo url dynamically also? because we are using almost 300 repos and im not sure how to configure it

No, not that I known of.

[thiDucTran]

@mars-esther our argocd notification template ...we pass it in as arguments to our template . including repo url, env, webhook id, etc... see below

apiVersion: v1
data:
  context: |
    argocdUrl:
  service.webhook.devlake: |
    url: http://apache-devlake-ui.infra.svc:4000
    headers:
    - name: Authorization
      value: Basic $devlake-basic-auth
  template.devlake-commit-status: |
    webhook:
      devlake:
        method: POST
        path: "/api/plugins/webhook/connections/{{.app.metadata.annotations.webhook}}/deployments"
        body: |
          {
            {{if eq .app.status.health.status "Healthy"}} "result": "SUCCESS"{{end}}
            {{if ne .app.status.health.status "Healthy"}} "result": "FAILURE"{{end}},
            "id": "{{.app.metadata.annotations.pipeline}}",
            "startedDate": "{{.app.status.operationState.startedAt}}",
            "finishedDate": "{{.app.status.operationState.finishedAt}}",
            "environment": "{{.app.metadata.annotations.env}}",
            "deploymentCommits":[
              {
                "startedDate": "{{.app.status.operationState.startedAt}}",
                "finishedDate": "{{.app.status.operationState.finishedAt}}",
                "repoUrl": "{{.app.metadata.annotations.repo}}",
                "commitSha": "{{.app.metadata.annotations.sha}}"
              }
            ]
          }
  trigger.on-deployed: |
    - when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status != ''
      send: [devlake-commit-status]
  trigger.on-health-degraded: |
    - when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status != ''
      send: [devlake-commit-status]
  trigger.on-sync-succeeded: |
    - when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status != ''
      send: [devlake-commit-status]
kind: ConfigMap

one of our argocd application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dora-python-app-dev
  namespace: default
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  annotations:
    sha: 'ce7038198a4639dcaacce3cf076ebdf65c438641'
    env: 'DEVELOPMENT'
    pipeline: 'dora-python-app-dev'
    repo: 'https://dev.azure.com/blah/foobar/_git/thi-test-dora-metrics'
    webhook: '1'
    notifications.argoproj.io/subscribe.on-deployed.devlake: ""
    notifications.argoproj.io/subscribe.on-health-degraded.devlake: ""
    notifications.argoproj.io/subscribe.on-sync-succeeded.devlake: ""
spec:
  destination:
    name: aks-rd-occluster-dev
    namespace: some-namespace
  project: default
  source:
    path: ./argocd-files/overlays/legacy-dev
    repoURL: https://dev.azure.com/blah/foobar/_git/thi-test-dora-metrics
    targetRevision: 'main'
  syncPolicy:
    automated:
      prune: true

[mars-esther]

@thiDucTran thank you so much for your elaborated answer. Since i am new to devlake, im not sure how to verify if the data has been sent to devlake . Is there a way to verify the data received from argocd in devlake

[DrFaust92]

Can confirm thiDucTran example works (with minor changes).

Worth to add this to the docs for webhook like circleCi has - https://devlake.apache.org/docs/Plugins/webhook/

[klesh]

PRs are welcome

[klesh]

@DrFaust92 Would you like to put up a PR to update the docs? Thanks in advance.

[testRedlink]

I'm also interested in the ArgoCD connection.

[DrFaust92]

klesh yes, ill try to get to it sometime this week

[klesh]

@DrFaust92 Nice, looking forward to it.