App Framework: it is not working with private buckets.

[yaroslav-nakonechnikov]

Please select the type of request

Bug

Tell us more

Describe the request splunk-operator application can't be configured with custom apps, if they are stored in private buckets and where it is forbidden to use secret keys.

Expected behavior If there is no possibility attach AWS policy to pod, inctance policy should be used instead. everything should work with s3://bucket_name notation on native way.

Splunk setup on K8S EKS 1.24 splunk-operator 2.1.1 splunk 9.0.3

Reproduction/Testing steps try any of next configuration:

"appRepo" = {
        "appsRepoPollIntervalSeconds" = "86400"
        "appSources" = [
          {
            "location"   = "config-explorer_1715.tgz"
            "volumeName" = "splunk-apps"
            "scope"      = "local"
            "name"       = "Config Explorer"
          }
        ]
        "volumes" = [{
          "endpoint"    = "https://splunk-operator-<AccountID>.s3-accesspoint.eu-central-1.amazonaws.com/"
          "name"        = "splunk-apps"
          "path"        = "splunk-apps/"
          "provider"    = "aws"
          "storageType" = "s3"
          "region"      = "eu-central-1"
        }]
      }

"appRepo" = {
        "appsRepoPollIntervalSeconds" = "86400"
        "appSources" = [
          {
            "location"   = "config-explorer_1715.tgz"
            "volumeName" = "splunk-apps"
            "scope"      = "local"
            "name"       = "Config Explorer"
          }
        ]
        "volumes" = [{
          "endpoint"    = "s3://splunk-operators-bucket"
          "name"        = "splunk-apps"
          "path"        = "splunk-apps/"
          "provider"    = "aws"
          "storageType" = "s3"
          "region"      = "eu-central-1"
        }]
      }

in all ways it drops logs like:

2023-01-17T09:49:22.534992739Z  ERROR   GetAppsList     Unable to list items in bucket  {"controller": "licensemanager", "controllerGroup": "enterprise.splunk.com", "controllerKind": "LicenseManager", "LicenseManager": {"name":"lm","namespace":"splunk-operator"}, "namespace": "splunk-operator", "name": "lm", "reconcileID": "216e03a6-2944-4e04-b75f-2afaca181255", "AWS S3 Bucket": "splunk-apps", "error": "AccessDenied: Access Denied\n\tstatus code: 403, request id: VPP022WAHETK9CCC, host id: dx2l8TLjNcD+0NNOEI8devQo9TdPYvF3TZGhR5UGkAxVQ/H40TmbME03vM9zLDOTHhijTiX5TLL="}

in our setup we can't use public access for buckets at all, as well as we can't use secret keys as authentication.

[yaroslav-nakonechnikov]

also, helm chart doesn't allow to pass env variables to splunk-operator, so it is not easy to add env vars in manager container, like:

   AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           eu-central-1
      AWS_REGION:                   eu-central-1
      AWS_ROLE_ARN:                 arn:aws:iam::AccountID:role/name
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token

[yaroslav-nakonechnikov]

another log:

023-01-17T13:46:33.187388246Z  ERROR   GetAppsList     Unable to list items in bucket  {"controller": "licensemanager", "controllerGroup": "enterprise.splunk.com", "controllerKind": "LicenseManager", "LicenseManager": {"name":"lm","namespace":"splunk-operator"}, "namespace": "splunk-operator", "name": "lm", "reconcileID": "0ec6ec50-80e2-45bc-beec-9aebcced1403", "AWS S3 Bucket": "splunk-apps", "error": "BucketRegionError: incorrect region, the bucket is not in 'eu-central-1' region at endpoint '', bucket is in 'us-east-1' region\n\tstatus code: 301, request id: 9SHNKVVZDKFBS1PC, host id: cj9lItWg+Rh9qKP4Vw58rHiOHyjZHSvVtm3FKaVlOFu+pZJ/AbXYza+0+Y14haaIatH/h1nbK91kB7m3mlnjLA=="}

config:

"appRepo" = {
        "appsRepoPollIntervalSeconds" = "86400"
        "appSources" = [
          {
            "location"   = "config-explorer_1715.tgz"
            "volumeName" = "splunk-apps"
            "scope"      = "local"
            "name"       = "Config Explorer"
          }
        ]
        "volumes" = [{
          "endpoint"    = "https://splunk-operator-AccountID.s3-accesspoint.eu-central-1.amazonaws.com/"
          "name"        = "splunk-apps"
          "path"        = "splunk-apps/"
          "provider"    = "aws"
          "storageType" = "s3"
          "region"      = "eu-central-1"
        }]

why it thinks that access point in different region?

[vivekr-splunk]

AWS IAM Service account is already supported in Splunk Operator. For Splunkd we are working add that feature. for splunk operator please follow the AWS documentation to add IAM service account to splunk-operator-controller-manager . please follow the steps given here https://aws.amazon.com/blogs/containers/diving-into-iam-roles-for-service-accounts/

example are here

> eksctl utils associate-iam-oidc-provider --region=us-west-2 --cluster=vivek --approve
> eksctl create iamserviceaccount --name splunk-operator-controller-manager --namespace splunk-operator --cluster vivek --role-name "oidc-test-role"  --attach-policy-arn arn:aws:iam::667741767953:policy/oidc-test-policy --approve --override-existing-serviceaccounts

by adding the IAM service account, it automatically adds environment variables

AWS_ROLE_ARN:                 arn:aws:iam::AccountID:role/name
AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token

let me know if you need any further clarifications.

[yaroslav-nakonechnikov]

splunk-operator - maybe. but it started to work only when i updated instance profile where splunk-operator was running.

anyway, still issues with using formats: aws allows at least 3 notations + accesspoints. Splunk operator supports only single way.

[vivekr-splunk]

@iaroslav-nakonechnikov we are working on supporting OIDC using AWS IAM serviceaccount for splunkd. will update you once it is supported.

[akondur]

Hi @iaroslav-nakonechnikov , please let us know if this issue of the instance profile has been resolved.

Could you elaborate on the different formats for AWS configurations with examples so that we can take a further look?

[yaroslav-nakonechnikov]

@akondur yes, now i see that all works as expected, and when app can't be downloaded - correct log message is shown.

For iam policy i'm not ready to answer it, atm i'm using it with node-profile, and it is fine. But later we will move to pod service account.

About ways to access bucket: https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-bucket-intro.html

• https://bucket-name.s3.region-code.amazonaws.com/key-name
• https://s3.region-code.amazonaws.com/bucket-name/key-name
• s3://bucket-name/key-name

all these 3 define same thing.

also, atm there is no support for private access endpoint, like: https://AccessPointName-AccountId.s3-accesspoint.region.amazonaws.com which may be required by policy not to use public endpoints.

[yaroslav-nakonechnikov]

this one becoming critical. splunk-operator itself should start to work with pod assigned role, otherwise system becomes non-complaint and requires to run splunk-operator controller on dedicated node.