Open gallettilance opened 1 year ago
@gallettilance A couple of questions:
Secret
, but in the example you are reading env
variables, it's not clear which approach you are asking for, can you clarify?Since this would potentially need changes in the helm
flow and the bundle/olm
generation flow (and corresponding runtime changes) , to add another authentication/authorization mechanism, I think this change needs larger discussion with the core maintainers.
OpenShift comes with the Cloud Credentials Operator (CCO). It provides a cloud agnostic experience for authenticating with cloud providers by adding a secret in the desired location with the credentials or config to get credentials for the operator to consume. Most if not all operators on OpenShift rely on CCO and hence rely on the secret created by it.
In the above example I attempted to set the env var to be the path to the config (at the mount path of the secret - /var/config
in the above example. My hope was that this would be enough for the controller to assume the role with web identity and receive STS credentials but I kept running into the following error when I create a Bucket
CR:
"error":"NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
Here is a diagram for how I would like this to work
@gallettilance I'm familiar with CCO, I don't think most operators use CCO. Unless you are talking about the cluster operators that already come with OpenShift, then yes those are going to be backed by CCO. To my knowledge, no certified or community operator uses this approach, but it could be one that I am unaware of.
With that said, ACK supports IRSA in OpenShift (Technically ROSA, since that's the only place OIDC is fully supported to my knowledge). Why isn't the IRSA approach sufficient?
I'm still not clear if you want this as an env
or as a secret
reference in the deployment. In OpenShift the CSV
contains both env
references and secret
references, which approach are you suggesting here? (sample spec below)
spec:
containers:
- args:
- --aws-region
- $(AWS_REGION)
- --aws-endpoint-url
- $(AWS_ENDPOINT_URL)
- --enable-development-logging
- $(ACK_ENABLE_DEVELOPMENT_LOGGING)
- --log-level
- $(ACK_LOG_LEVEL)
- --resource-tags
- $(ACK_RESOURCE_TAGS)
- --watch-namespace
- $(ACK_WATCH_NAMESPACE)
command:
- ./bin/controller
env:
- name: ACK_SYSTEM_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
envFrom:
- configMapRef:
name: ack-s3-user-config
optional: false
- secretRef:
name: ack-s3-user-secrets
optional: true
IRSA is insufficient because it depends on the pod identity webhook which only supports AWS. CCO aims to support all cloud providers.
What does:
envFrom:
- configMapRef:
name: ack-s3-user-config
optional: false
- secretRef:
name: ack-s3-user-secrets
optional: true
do and what do each of these configmap and secrets look like? (where can I read more about this?) Because maybe I just need to add the config from the secret to a configMap for the pod to consume.
I just want the ACK controller to use the config specified in the secret. This can be by including the secret ref in an env var or any other way (I'm open to suggestions). I just thought setting the AWS_CONFIG_FILE to the config path would be sufficient but it's not - and I'm not sure I understand why.
@gallettilance Here are the docs on how to install on OpenShift. The existence of these is so that a user can provide configuration: ConfigMap
for how to run the controller and Secret
containing auth information so the controller can authenticate with AWS, if using an access key, that's why this is optional.
These are envFrom
since a cluster admin cannot modify a Deployment
after it has been installed by OLM, since OLM will reconcile the Deployment
and the changes they made will be lost. I hope this is clearer on what is going on.
Sure - but in this case I am creating / customizing the CSV so I have full control of the deployment. All I want is to give the controller the above config so that it can assume the role with web identity and get STS creds. Are you saying if I just put the contents of the above secret in a configmap, the ack controller will be able to use STS creds? If not where is this config suppose to go?
Or are you saying that the ack controller doesn't support this approach - the only way to get sts creds is to annotate the service account with the role ARN and have the pod identity webhook project the bound sa token into the pod?
Or can I use the above secret
in lieu of one that is supposed to contain credentials information?
(sorry for all the questions I'm just having a hard time understanding)
I also use this model in my AWS Assume Role Operator and it works with the AWS Load Balancer Controller, which is not part of ACK. However, the next problem I face is that when the token is refreshed in the secret, it is not reloaded, not by the Load Balancer Controller anyway. This should also be checked with the ACK controllers, otherwise the model won't work.
Issues go stale after 180d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 60d of inactivity and eventually close.
If this issue is safe to close now please do so with /close
.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale
Issues go stale after 180d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 60d of inactivity and eventually close.
If this issue is safe to close now please do so with /close
.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale
Is your feature request related to a problem?
Background
Operators (especially on OpenShift) that need to authenticate with the cloud provider, have started using the following flow to support authenticating using STS:
The issue
This flow does not seem to work with ACK controllers. Following the docs, here is how I tried to achieve this:
/some/location
AWS_CONFIG_FILE
env var to/some/location
(I also tried setting theAWS_SHARED_CREDENTIALS_FILE
to that same location)Describe the solution you'd like
Option 1
Setting either the
AWS_CONFIG_FILE
or theAWS_SHARED_CREDENTIALS_FILE
to the mount path location that contains the above info should be sufficient for the ACK controller to assume the role and receive STS creds.Option 2
It is possible for the ACK controller to read the secret directly from the apiserver.
Some more info
I configured things properly on my cluster and have an OIDC provider and role created with the right permissions for the ack controller I'm trying to install (specifically was using the s3 one). I know things are set up right because I used the pod identity webhook approach by annotating the service account and that worked seamlessly. I also verified that the secret is properly mounted to the pod and the file is readable, contains the correct info, and the web identity token is valid.
Here is the CSV I used to install the controller:
quay.io/lgallett/s3:latest
is built from the s3-controller repo.