App Framework: ValidateAppFrameworkSpec fails for maxConcurrentAppDownloads

[PaintedShepherd]

Please select the type of request

Bug

Tell us more

Describe the request

• We have installed an splunk operator followed by an enterprise deployment.
• Unfortunately since we enabled the app repos, to download apps from our s3-bucket, the operator is throwing the following error:

We have an OIDC provider to use an iam-role with a policy to access the s3-bucket. We also added a remote access key, because the operator complained, that there is no secretRef set.

Error Log of Splunk Operator

2024-01-09T13:23:36.742227343Z  INFO    Starting workers    {"controller": "monitoringconsole", "controllerGroup": "enterprise.splunk.com", "controllerKind": "MonitoringConsole", "worker count": 15}
2024-01-09T13:23:36.742393296Z  INFO    start   {"controller": "monitoringconsole", "controllerGroup": "enterprise.splunk.com", "controllerKind": "MonitoringConsole", "MonitoringConsole": {"name":"mc","namespace":"splunk"}, "namespace": "splunk", "name": "mc", "reconcileID": "523eeb55-ac23-4c8f-b518-802a890aeccf", "monitoringconsole": "splunk/mc", "CR version": "1036624"}
2024-01-09T13:23:36.742429568Z  INFO    ValidateAppFrameworkSpec    configCheck {"controller": "monitoringconsole", "controllerGroup": "enterprise.splunk.com", "controllerKind": "MonitoringConsole", "MonitoringConsole": {"name":"mc","namespace":"splunk"}, "namespace": "splunk", "name": "mc", "reconcileID": "523eeb55-ac23-4c8f-b518-802a890aeccf", "scope": true}
2024-01-09T13:23:36.742445208Z  INFO    ValidateAppFrameworkSpec    Invalid value of maxConcurrentAppDownloads  {"controller": "monitoringconsole", "controllerGroup": "enterprise.splunk.com", "controllerKind": "MonitoringConsole", "MonitoringConsole": {"name":"mc","namespace":"splunk"}, "namespace": "splunk", "name": "mc", "reconcileID": "523eeb55-ac23-4c8f-b518-802a890aeccf", "configured value": 0, "Setting it to default value": 5}
2024-01-09T13:23:36.742503399Z  INFO    Observed a panic in reconciler: assignment to entry in nil map  {"controller": "monitoringconsole", "controllerGroup": "enterprise.splunk.com", "controllerKind": "MonitoringConsole", "MonitoringConsole": {"name":"mc","namespace":"splunk"}, "namespace": "splunk", "name": "mc", "reconcileID": "523eeb55-ac23-4c8f-b518-802a890aeccf"}
panic: assignment to entry in nil map [recovered]
    panic: assignment to entry in nil map

goroutine 786 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:119 +0x1e5
panic({0x19b6fe0?, 0x2046e50?})
    /usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/splunk/splunk-operator/pkg/splunk/enterprise.validateSplunkAppSources(0xc000aba230, 0x1, {0x184b609, 0x11})
    /workspace/pkg/splunk/enterprise/configuration.go:1474 +0x305
github.com/splunk/splunk-operator/pkg/splunk/enterprise.ValidateAppFrameworkSpec({0x2065858, 0xc000017ef0}, 0xc000aba230, 0xc000aba2f8, 0x1, {0x184b609, 0x11})
    /workspace/pkg/splunk/enterprise/configuration.go:1575 +0x72f
github.com/splunk/splunk-operator/pkg/splunk/enterprise.validateMonitoringConsoleSpec({0x2065858, 0xc000017ef0}, {0x7f27ac05c688, 0xc0009958c0}, 0xc000ab9b00)
    /workspace/pkg/splunk/enterprise/monitoringconsole.go:212 +0x137
github.com/splunk/splunk-operator/pkg/splunk/enterprise.ApplyMonitoringConsole({0x2065858, 0xc000017ef0}, {0x7f27ac05c688, 0xc0009958c0}, 0xc000ab9b00)
    /workspace/pkg/splunk/enterprise/monitoringconsole.go:58 +0x1ad
github.com/splunk/splunk-operator/controllers.glob..func6({0x2065858, 0xc000017ef0}, {0x206d598?, 0xc0009958c0}, 0x5?)
    /workspace/controllers/monitoringconsole_controller.go:114 +0x52
github.com/splunk/splunk-operator/controllers.(*MonitoringConsoleReconciler).Reconcile(0xc000a1a738, {0x2065858, 0xc000017ef0}, {{{0xc000aa2cc8, 0x6}, {0xc000aa2cc6, 0x2}}})
    /workspace/controllers/monitoringconsole_controller.go:104 +0x55b

Expected behavior

• No more error and apps will be installed from our private and encrypted s3-bucket.

Splunk setup on K8S

• Details of the Splunk setup on the K8s cluster.

This is our values.yaml of the splunk operator and the enterprise deployment. Note that it is converted to typescript/json, because we are deploying with AWS CDK.

Splunk Operator Values Override

        image: {
          repository: 'REPONAME/splunk/splunk:' + (props.splunkImageVersion ?? '9.1.1'),
        },
        kubeRbacProxy: {
          image: {
            repository: 'REPONAME/kubebuilder/kube-rbac-proxy',
            tag: props.rbacImageVersion ?? 'v0.13.1',
          },
        },
        splunkOperator: {
          image: {
            repository: 'REPONAME/splunk/splunk-operator:' + (props.splunkOperatorVersion ?? '2.4.0'),
          },
          annotations:{
            "eks.amazonaws.com/role-arn": serviceAccount.role.roleArn
          },
          podAnnotations: {
            "eks.amazonaws.com/role-arn": serviceAccount.role.roleArn
          }
        },

Splunk Enterprise Values Override

        image: {
          repository: 'REPONAME/splunk/splunk:'+ (props.splunkImageVersion ?? '9.1.1'),
        },
        'splunk-operator': {
          enabled: false,
        },
        clusterManager: {
          enabled: true,
          serviceAccount: serviceAccount.serviceAccountName,
          appRepo:
          {
            appsRepoPollIntervalSeconds: 900,
            defaults:
            {
              volumeName: "volume_app_repo_zlom-sandbox_idx",
              scope: "cluster"
            },
            appSources:
              [
                {
                  name: "platform-apps",
                  location: "platform-apps/"
                },
                {
                  name: "consumer-apps",
                  location: "consumer-apps/"
                }
              ],
            volumes: [
              {
                name: "volume_app_repo_zlom-sandbox_idx",
                storageType: "s3",
                provider: "aws",
                path: "our-s3-bucket/idx/",
                endpoint: "https://s3-eu-central-1.amazonaws.com",
                region: "eu-central-1",
                secretRef: "s3-secret"
              }
            ],
          },
          etcVolumeStorageConfig: {
            storageClassName: 'ebs-csi-default',
          },
          varVolumeStorageConfig: {
            storageClassName: 'ebs-csi-default',
          },
          licenseUrl: 'LICENSE_SERVER',
          defaults: {
            splunk: {
              idxc: {
                replication_factor: 2,
                search_factor: 2,
              },
            },
          },
          resources: {
            requests: {
              memory: '4Gi',
              cpu: '2',
            },
            limits: {
              memory: '4Gi',
              cpu: '2',
            },
          },
        },
        indexerCluster: {
          enabled: true,
          serviceAccount: serviceAccount.serviceAccountName,
          replicaCount: 1,
          licenseUrl: 'LICENSE_SERVER',
          etcVolumeStorageConfig: {
            ephemeralStorage: false,
            storageCapacity: '10Gi',
            storageClassName: 'ebs-csi-default',
          },
          varVolumeStorageConfig: {
            ephemeralStorage: false,
            storageCapacity: '100Gi',
            storageClassName: 'ebs-csi-default',
          },
          resources: {
            requests: {
              memory: '12Gi',
              cpu: '6',
            },
            limits: {
              memory: '12Gi',
              cpu: '6',
            },
          },
        },
        searchHeadCluster: {
          enabled: true,
          serviceAccount: serviceAccount.serviceAccountName,
          appRepo:
          {
            appsRepoPollIntervalSeconds: 900,
            defaults:
            {
              volumeName: "volume_app_repo_zlom-sandbox_sh",
              scope: "cluster"
            },
            appSources:
              [{
                name: "platform-apps",
                location: "platform-apps/"
              },
              {
                name: "consumer-apps",
                location: "consumer-apps/"
              }],
            volumes:
              [{
                name: "volume_app_repo_zlom-sandbox_sh",
                storageType: "s3",
                provider: "aws",
                path: "our-s3-bucket/sh/",
                endpoint: "https://s3-eu-central-1.amazonaws.com",
                region: "eu-central-1",
                secretRef: "s3-secret"
              }],
          },
          resources: {
            requests: {
              memory: '12Gi',
              cpu: '6',
            },
            limits: {
              memory: '12Gi',
              cpu: '6',
            },
          },
        },
        monitoringConsole: {
          enabled: true,
          appRepo:
          {
            appsRepoPollIntervalSeconds: 900,
            defaults:
            {
              volumeName: "volume_app_repo_zlom-sandbox_mc",
              scope: "cluster"
            },
            appSources:
              [{
                name: "platform-apps",
                location: "platform-apps/"
              },
              {
                name: "consumer-apps",
                location: "consumer-apps/"
              }],
            volumes:
              [{
                name: "volume_app_repo_zlom-sandbox_mc",
                storageType: "s3",
                provider: "aws",
                path: "our-s3-bucket/mc/",
                endpoint: "https://s3-eu-central-1.amazonaws.com",
                region: "eu-central-1",
                secretRef: "s3-secret"
              }],
          },        
        },
        standalone: {
          enabled: true,
          name: "hf",
          serviceAccount: serviceAccount.serviceAccountName,
          replicaCount: 2,
          appRepo:
          {
            appsRepoPollIntervalSeconds: 900,
            defaults:
            {
              volumeName: "volume_app_repo_zlom-sandbox_hf",
              scope: "cluster"
            },
            appSources:
              [{
                name: "platform-apps",
                location: "platform-apps/"
              },
              {
                name: "consumer-apps",
                location: "consumer-apps/"
              }],
            volumes:
              [{
                name: "volume_app_repo_zlom-sandbox_hf",
                storageType: "s3",
                provider: "aws",
                path: "our-s3-bucket/hf/",
                endpoint: "https://s3-eu-central-1.amazonaws.com",
                region: "eu-central-1",
                secretRef: "s3-secret"
              }],
          },
          licenseUrl: 'LICENSE_SERVER',
          etcVolumeStorageConfig: {
            storageCapacity: '10Gi',
            storageClassName: 'ebs-csi-default',
          },
          varVolumeStorageConfig: {
            storageCapacity: '100Gi',
            storageClassName: 'ebs-csi-default',
          },
          resources: {
            requests: {
              memory: '8Gi',
              cpu: '4',
            },
            limits: {
              memory: '8Gi',
              cpu: '4',
            },
          },
        },

Reproduction/Testing steps

• install the operator and the splunk enterprise deployment,
• activate the app repo in SH, IDX, STNDL and CM
• create and add a service account to the sh, idx, stndl and cm via enterprise deployment values.yml
• add annotation and podAnnotation like we did in the values of the splunk operator
• add apps to the s3 bucket
• watch the operator logs of the manager container

K8s environment

• Useful information about the K8S environment being used. Eg. version of K8s, kind of K8s cluster etc..
• Version 1.26
• Operator Version 2.4.0
• Splunk Version: 9.1.1

Additional context(optional)

• If you need anything else, let me know.

[PaintedShepherd]

I found the issue myself. It was the wrong scope of the app repos. The MC and he SNTLN app repos has to be "local" instead of "cluster". Hope this helps someone.