Bundle is not removed after removing GitRepo configured with multiple paths

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

After deleting GitRepo pointing to multiple paths(Service and ConfigMap), configmap getting removed but service is not being removed instead it is re-created. Also, observed that bundle not removed after deleting GitRepo, may causing this issue. Same service app is created under Apps which is being re-created again and again.

Expected Behavior

Once GitRepo removed no leftover should be present.

Steps To Reproduce

Create a GitRepo which is pointing to multiple paths. (I used: https://github.com/rancher/fleet-test-data and paths: multiple-paths)
Wait for applications to deployed on each cluster (if there are more than one)
Now delete GitRepo
Check the applications are removed correctly or not.
Check the bundle it is still there and didn't get removed.

Environment

- Architecture:
- Fleet Version:fleet:v0.10.0-rc.18
- Cluster:
  - Provider:k3s
  - Options:
  - Kubernetes Version:

Logs

GitJob Logs

``` {"level":"info","ts":"2024-07-03T09:57:14Z","logger":"git-latest-commit-poll-watch","msg":"new commit found","gitrepo":{"kind":"GitRepo","apiVersion":"fleet.cattle.io/v1alpha1","metadata":{"name":"correct-drift-service-test","namespace":"fleet-default","uid":"a424abd2-32f3-4c3e-a623-45f76e1c95c2","resourceVersion":"16236","generation":1,"creationTimestamp":"2024-07-03T09:57:14Z","managedFields":[{"manager":"rancher","operation":"Update","apiVersion":"fleet.cattle.io/v1alpha1","time":"2024-07-03T09:57:14Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{".":{},"f:branch":{},"f:correctDrift":{".":{},"f:enabled":{}},"f:insecureSkipTLSVerify":{},"f:paths":{},"f:repo":{},"f:targets":{}}}}]},"spec":{"repo":"https://github.com/rancher/fleet-test-data","branch":"master","paths":["multiple-paths"],"targets":[{"clusterSelector":{"matchExpressions":[{"key":"provider.cattle.io","operator":"NotIn","values":["harvester"]}]}}],"imageScanCommit":{"authorName":"","authorEmail":""},"correctDrift":{}},"status":{"observedGeneration":0,"readyClusters":0,"desiredReadyClusters":0,"summary":{"ready":0,"desiredReady":0},"display":{},"resourceCounts":{"ready":0,"desiredReady":0,"waitApplied":0,"modified":0,"orphaned":0,"missing":0,"unknown":0,"notReady":0},"lastSyncedImageScanTime":null}},"commit":"4766e074c36c2820ea792420ad35ec1c3931c0e3"} {"level":"info","ts":"2024-07-03T09:57:14Z","logger":"gitjob","msg":"job deletion triggered because of generation change","controller":"gitrepo","controllerGroup":"fleet.cattle.io","controllerKind":"GitRepo","GitRepo":{"name":"correct-drift-service-test","namespace":"fleet-default"},"namespace":"fleet-default","name":"correct-drift-service-test","reconcileID":"78d3c2e4-95d4-4181-add4-3f630d3342b9","generation":2,"commit":"4766e074c36c2820ea792420ad35ec1c3931c0e3"} {"level":"info","ts":"2024-07-03T09:57:27Z","logger":"gitjob","msg":"job deletion triggered because of generation change","controller":"gitrepo","controllerGroup":"fleet.cattle.io","controllerKind":"GitRepo","GitRepo":{"name":"correct-drift-service-test","namespace":"fleet-default"},"namespace":"fleet-default","name":"correct-drift-service-test","reconcileID":"bf995b4b-ef0c-4851-882f-abb07b827905","generation":3,"commit":"4766e074c36c2820ea792420ad35ec1c3931c0e3"} {"level":"info","ts":"2024-07-03T09:57:27Z","logger":"gitjob","msg":"job deletion triggered because of generation change","controller":"gitrepo","controllerGroup":"fleet.cattle.io","controllerKind":"GitRepo","GitRepo":{"name":"correct-drift-service-test","namespace":"fleet-default"},"namespace":"fleet-default","name":"correct-drift-service-test","reconcileID":"a51440a3-c576-4b62-a734-ce7a1538592a","generation":4,"commit":"4766e074c36c2820ea792420ad35ec1c3931c0e3"} {"level":"info","ts":"2024-07-03T09:57:42Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.resources[5].perClusterState[0].patch.spec\""} ```

Anything else?

In my GitRepo, ConfigMaps gets removed but Service remains as it is and re-created when try to delete it. Also, there is created for the same Service which cannot be removed.

I managed to recreate this one, but it was not happening all the time.

Adding here my resources just in case they're useful for QA.

I tested with the following GitRepo

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: sample
  namespace: fleet-local
spec:
  repo: "https://github.com/rancher/fleet-test-data"
  branch: master
  paths:
  - multiple-paths

and the following bash script:

#!/bin/bash

i=1

while true
do
  echo "---------------------------"
  kubectl apply -f test-2586.yaml

  OUT=""
  while [ "$OUT" != "2" ];
  do
   OUT=`kubectl get gitrepo sample -n fleet-local -o jsonpath='{.status.summary.ready}'`
  done

  kubectl delete gitrepo sample -n fleet-local
  kubectl wait --for=delete gitrepo/sample --timeout=60s -n fleet-local

  kubectl get bundles -A
  BUNDLES_LEFT=`kubectl get bundles -A | grep sample | wc -l`

  if [ "$BUNDLES_LEFT" == "       0" ];
  then
    echo "ALL GOOD attempt number: ${i}"
    echo "---------------------------"
    i=$(($i+1))
  else
    echo "FAILED"
    echo "---------------------------"
    exit 1
  fi
done

It fails repeatedly with the version without the fix:

---------------------------
gitrepo.fleet.cattle.io/sample created
gitrepo.fleet.cattle.io "sample" deleted
NAMESPACE     NAME                BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local   1/1
ALL GOOD attempt number: 8
---------------------------
---------------------------
gitrepo.fleet.cattle.io/sample created
gitrepo.fleet.cattle.io "sample" deleted
NAMESPACE     NAME                            BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local               1/1
fleet-local   sample-multiple-paths-service   1/1
FAILED
---------------------------

---------------------------
gitrepo.fleet.cattle.io/sample created
gitrepo.fleet.cattle.io "sample" deleted
NAMESPACE     NAME                BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local   1/1
ALL GOOD attempt number: 15
---------------------------
---------------------------
gitrepo.fleet.cattle.io/sample created
gitrepo.fleet.cattle.io "sample" deleted
NAMESPACE     NAME                            BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local               1/1
fleet-local   sample-multiple-paths-service   1/1
FAILED
---------------------------

and I was able to run the test 210 times and it did not fail just with the change in https://github.com/rancher/fleet/pull/2591

---------------------------
gitrepo.fleet.cattle.io/sample created
gitrepo.fleet.cattle.io "sample" deleted
NAMESPACE     NAME                BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local   1/1
ALL GOOD attempt number: 210
---------------------------

rancher / fleet