search

apache / camel-karavan

Apache Camel Karavan a Low-code Data Integration Platform
https://camel.apache.org
Apache License 2.0
447 stars 153 forks source link

Bug: karavan.git.repository doesn't work correct at 4.7.0 #1347

Closed drblack666 closed 1 month ago

drblack666 commented 2 months ago

Describe the bug

I tried to upgrade karavan from 4.5.1 to 4.7.0 in my k8s cluster and got an error. For some reason it uses gitea instead of my gitlab url. In pod logs:

INFO  [org.apa.cam.kar.ser.GitService] (main) Error connecting git: http://gitea:3000/karavan/karavan.git: cannot open git-upload-pack

The environment variable karavan.git.repository set up correctly:

root@karavan-8d998b4b6-gmh7b:/# env | grep git
karavan.git.repository=https://gitlab.<my_domain>.com

Downgrade to 4.6.0 or 4.5.1 fixed this problem

Steps to reproduce the behavior

Use custom karavan.git.repository and create kubernetes deployment with karavan version 4.7.0

Variant

Web Application

Container Management (if applicable)

Kubernetes

Operating System (if applicable)

None

Version

4.7.0

Relevant log output

2024-08-02 13:11:35,095 INFO  [org.apa.cam.kar.kub.KubernetesStatusService] (main) Status Listeners: starting...
2024-08-02 13:11:35,096 INFO  [org.apa.cam.kar.kub.KubernetesStatusService] (main) Stop Kubernetes Informers
2024-08-02 13:11:35,097 INFO  [org.apa.cam.kar.kub.KubernetesStatusService] (main) Starting Kubernetes Informers
2024-08-02 13:11:36,184 INFO  [org.apa.cam.kar.kub.KubernetesStatusService] (main) Started Kubernetes Informers
2024-08-02 13:11:36,184 INFO  [org.apa.cam.kar.kub.KubernetesStatusService] (main) Status Listeners: started
2024-08-02 13:11:36,188 INFO  [org.apa.cam.kar.KaravanStartupLoader] (main) Starting karavan in dev env in Kubernetes
2024-08-02 13:11:36,188 INFO  [org.apa.cam.kar.KaravanStartupLoader] (main) Projects loading...
2024-08-02 13:11:36,189 INFO  [org.apa.cam.kar.ser.GitService] (main) Check git
2024-08-02 13:11:36,194 INFO  [org.apa.cam.kar.ser.GitService] (main) Set UsernamePasswordCredentialsProvider
2024-08-02 13:11:39,357 INFO  [org.apa.cam.kar.ser.GitService] (main) Error connecting git: http://gitea:3000/karavan/karavan.git: cannot open git-upload-pack
2024-08-02 13:11:39,357 INFO  [org.apa.cam.kar.KaravanStartupLoader] (main) Starting Project service: git is ready
mgubaidullin commented 2 months ago

Could you please try with KARAVAN_GIT_REPOSITORY instead of karavan.git.repository https://github.com/quarkusio/quarkus/wiki/Migration-Guide-3.5#kubernetes-environment-variables

drblack666 commented 2 months ago

Yeah, now it sees correct repository and according to the logs writes on the correct branch, but I can't see anything. The repository is empty. If I change image version to 4.6.0 it works fine

2024-08-05 06:16:46,136 INFO  [org.apa.cam.kar.lis.CommitListener] (executor-thread-14) Commit event: {
  "message" : "newproj",
  "projectId" : "newproj",
  "userId" : "fb1fd1b0-702d-4daa-a149-a0fea5ee54ad"
}
2024-08-05 06:16:46,136 INFO  [org.apa.cam.kar.ser.ProjectService] (executor-thread-14) Commit project: newproj
2024-08-05 06:16:46,137 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Commit and push project newproj
2024-08-05 06:16:46,137 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Temp folder created /tmp/bbdb8efb-93d9-44de-b698-6381014de26b13943963097886175367
2024-08-05 06:16:46,137 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Git checkout https://<gitlab_domain>/karavan-configuration
2024-08-05 06:16:46,137 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Temp folder created /tmp/bbdb8efb-93d9-44de-b698-6381014de26b13943963097886175367
2024-08-05 06:16:46,137 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Set UsernamePasswordCredentialsProvider
2024-08-05 06:16:46,178 INFO  [org.ecl.jgi.tra.TransportHttp] (executor-thread-14) URI 'https://<gitlab_domain>/karavan-configuration': following HTTP redirect #0  https://<gitlab_domain>/karavan-configuration/ -> https://<gitlab_domain>/karavan-configuration.git/
2024-08-05 06:16:46,531 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Write files for project newproj
2024-08-05 06:16:46,532 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Add file deployment.jkube.yaml
2024-08-05 06:16:46,532 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Add file application.properties
2024-08-05 06:16:46,532 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Add deleted files to git index for project newproj
2024-08-05 06:16:46,532 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Checking file application.properties
2024-08-05 06:16:46,532 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Checking file deployment.jkube.yaml
2024-08-05 06:16:46,532 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Commit and push changes to the branch master
2024-08-05 06:16:46,535 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Git add: org.eclipse.jgit.dircache.DirCache@7fd1ed39
2024-08-05 06:16:46,539 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Git commit: commit 3f078a3d6cecd883364979af159285556050eb61 1722838606 ------p
2024-08-05 06:16:46,539 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Set UsernamePasswordCredentialsProvider
2024-08-05 06:16:46,579 INFO  [org.ecl.jgi.tra.TransportHttp] (executor-thread-14) URI 'https://<gitlab_domain>/karavan-configuration': following HTTP redirect #0  https://<gitlab_domain>/karavan-configuration/ -> https://<gitlab_domain>/karavan-configuration.git/
2024-08-05 06:16:46,759 INFO  [org.apa.cam.kar.ser.GitService] (executor-thread-14) Git push: [org.eclipse.jgit.transport.PushResult@51852f2c]
enter-marlah commented 2 months ago

We have a similar problem.

First we replaced the karavan.git.repository with KARAVAN_GIT_REPOSITORY in the deployment.yaml to get the git connection to working.

The problem now is with the git branch. Karavan always uses main branch and pushes to the branch are not working.

We replaced karavan.git.branch with KARAVAN_GIT_BRANCH with no luck. Karavan always reads from the main branch and when we do a push logs say its using the correct branch but nothing is coming to git. The pushes dont appear in main branch or in the custom branch.

mgubaidullin commented 2 months ago

Please share your deployment.yaml

enter-marlah commented 2 months ago

Here is the contents of the deployment.yaml

---
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
  name: "_CI_BRANCH_NAME_"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "_CI_BRANCH_NAME_"
  template:
    metadata:
      labels:
        app: "_CI_BRANCH_NAME_"
    spec:
      containers:
      - env:
        - name: "KARAVAN_DEVMODE_SERVICE_ACCOUNT"
          value: "karavan"
        - name: "KARAVAN_BUILDER_SERVICE_ACCOUNT"
          value: "karavan"
        - name: "KARAVAN_ENVIRONMENT"
          value: "dev"          
        - name: "KARAVAN_CONTAINER_STATUS_INTERVAL"
          value: "disabled"
        - name: "KARAVAN_CONTAINER_STATISTICS_INTERVAL"
          value: "disabled"
        - name: "KARAVAN_CAMEL_STATUS_INTERVAL"
          value: "3s"
        - name: "KARAVAN_DEVMODE_IMAGE"
          value: "ghcr.io/apache/camel-karavan-devmode:4.7.0"
        - name: "KARAVAN_GIT_REPOSITORY"
          valueFrom:
            secretKeyRef:
              key: "karavan.git.repository"
              name: "karavan"
        - name: "KARAVAN_GIT_USERNAME"
          valueFrom:
            secretKeyRef:
              key: "karavan.git.username"
              name: "karavan"
        - name: "KARAVAN_GIT_PASSWORD"
          valueFrom:
            secretKeyRef:
              key: "karavan.git.password"
              name: "karavan"
        - name: "KARAVAN_GIT_BRANCH"
          valueFrom:
            secretKeyRef:
              key: "karavan.git.branch"
              name: "karavan"
        - name: "KARAVAN_KEYCLOAK_URL"
          valueFrom:
            secretKeyRef:
              key: "karavan.keycloak.url"
              name: "karavan"
        - name: "KARAVAN_KEYCLOAK_REALM"
          valueFrom:
            secretKeyRef:
              key: "karavan.keycloak.realm"
              name: "karavan"
        - name: "KARAVAN_KEYCLOAK_FRONTEND_CLIENTID"
          valueFrom:
            secretKeyRef:
              key: "karavan.keycloak.frontend.clientId"
              name: "karavan"
        - name: "KARAVAN_KEYCLOAK_BACKEND_CLIENTID"
          valueFrom:
            secretKeyRef:
              key: "karavan.keycloak.backend.clientId"
              name: "karavan"
        - name: "KARAVAN_KEYCLOAK_BACKEND_SECRET"
          valueFrom:
            secretKeyRef:
              key: "karavan.keycloak.backend.secret"
              name: "karavan"
        - name: "KUBERNETES_NAMESPACE"
          valueFrom:
            fieldRef:
              apiVersion: ""
              fieldPath: "metadata.namespace"
        image: "ghcr.io/apache/camel-karavan:4.7.0"
        imagePullPolicy: "Always"
        name: "_CI_BRANCH_NAME_"
        ports:
        - containerPort: 8080
          name: "_CI_BRANCH_NAME_"
        resources:
          requests:
            memory: "1024Mi"
      serviceAccount: "karavan"
thgruiz commented 2 months ago

I think this would be fixed with https://github.com/apache/camel-karavan/pull/1389

enter-marlah commented 2 months ago

I tried the new deployment.yaml from the PR and the Karavan still uses the main branch regardless of the value in KARAVAN_GIT_BRANCH secret.

thgruiz commented 2 months ago

I tried the new deployment.yaml from the PR and the Karavan still uses the main branch regardless of the value in KARAVAN_GIT_BRANCH secret.

Yeah, sorry I missed the main reason of this bug report.

The PR I said above only chances the env var to upper case, but I did not change anything related to java code (that does the poll/push from git).

Note: my push was not working too, but after some debugging it was because my git repo's default branch was "master" and karavan-app tried to check out the inexistent branch and gives an error. Just until this is fixed, change the default branch of your repo to "main" and the pushes may work

t33muki commented 1 month ago

change the default branch of your repo to "main" and the pushes may work

Thanks, our default is main. Problem is that our specific branches get ignored. We deploy Karavans dynamically to different namespaces and use our 'main' branch as a base. Pipeline creates - say- 'test1' branch and adjusts Karavan secrets accordingly and deploys it to cluster. The KARAVAN_GIT_BRANCH secret now correctly contains 'test1' on the deployed Karavan - but the UI shows projects from the 'main' branch. On 4.6.0 the branches work as intended and ui shows contents of the correct branch.

When I compared the versions 4.6.0 and 4.7.0 of karavan-app/src/main/java/org/apache/camel/karavan/service/GitService.java I noticed the branch handling of the getGit-method had changed quite a bit - I unfortunately don't have the knowledge or the tools to debug it further.