search

scylladb / scylla-operator

The Kubernetes Operator for ScyllaDB
https://operator.docs.scylladb.com/
Apache License 2.0
331 stars 163 forks source link

Allow `extraVolumes`, `extraVolumeMounts` & `extraEnvs` in Scylla manager Agent helm charts #1395

Open Kavinraja-G opened 1 year ago

Kavinraja-G commented 1 year ago

Is this a bug report or feature request?

What should the feature do: I would like to have extraVolumes, extraVolumeMounts & extraEnvs in the helm chart.

What is use case behind this feature: This will allow the scylla-manager-agents to use the IRSA to authenticate with AWS services like S3 for backups. For example in anthos on AWS requires the mentioned steps to enable WLI.

Additional Information: Example helm values for scylla manager I'm expecting --

    extraEnv:
    - name: AWS_ROLE_ARN
      value: AWS_ROLE_ARN
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/aws-iam-token/serviceaccount/token
    - name: AWS_REGION
      value: AWS_REGION
    extraVolumeMounts:
    - mountPath: /var/run/secrets/aws-iam-token/serviceaccount
      name: aws-iam-token
      readOnly: true
    extraVolumes:
    - name: aws-iam-token
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            audience: sts.amazonaws.com
            expirationSeconds: 86400
            path: token
Kavinraja-G commented 1 year ago

Please let me know if anything I can help to enable this feature. TY!

scylla-operator-bot[bot] commented 2 months ago

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

/lifecycle stale

scylla-operator-bot[bot] commented 1 month ago

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

/lifecycle rotten

chidambaram27 commented 3 weeks ago

facing similar issue, scylla manager couldn't able to access buckets while performing backup restore action. Looking for a way to introduce workload identity env variables for scylla manager.

{"L":"INFO","T":"2024-08-21T08:33:59.708Z","N":"cluster.client","M":"HTTP retry backoff","operation":"OperationsList","wait":"1s","error":"agent [HTTP 500] error in ListJSON: WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: 65f9229d-e343-4457-83d5-398c11fc0a62","_trace_id":"JMhZ4tlTRGGJiCw8-yUlaQ"}

rzetelskik commented 1 week ago

@chidambaram27 service account token projection can be set up using a AWS shared config file (next in chain after the env variables, which can't be configured with ScyllaCluster API) with volumes and agentVolumeMounts:

You'd need to create a secret with the config file

config=$(cat << EOF
[default]
role_arn=AWS_ROLE_ARN
web_identity_token_file=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
region=AWS_REGION

EOF
)
kubectl -n scylla create secret generic aws-config --from-literal=config=${config}

The example ScyllaCluster configuration (limitied to the relevant parts) would be:

apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
metadata:
  name: scylla-cluster
spec:
  agentVersion: 3.3.0
  ...
  datacenter:
    name: us-east-1
    racks:
      - name: a
        agentVolumeMounts:
        - mountPath: /var/lib/scylla-manager/.aws/config
          subPath: config
          name: aws-config
          readOnly: true
        - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
          name: aws-iam-token
          readOnly: true
        volumes:
        - name: aws-config
          secret:
            secretName: aws-config
        - name: aws-iam-token
          projected:
            defaultMode: 420
            sources:
            - serviceAccountToken:
                audience: sts.amazonaws.com
                expirationSeconds: 86400
                path: token
        ...

When configuring the web identity remember to use the service account created by Scylla Operator for your ScyllaCluster, i.e. <ScyllaCluster name>-member.

Before you schedule the restore task you can test this as per https://manager.docs.scylladb.com/stable/backup/setup-amazon-s3.html#troubleshoot-connectivity.

chidambaram27 commented 1 week ago

yeah that's right, that's how I configured the scylla cluster. But the error that I mentioned before is from Scylla Manager itself. I hope it also needs the access to the backup location to schedule this restore task. That's where I got blocked on how to configure this aws shared config file as there is no provision for additional volume mount in Scylla Manager helm chart.