Openfaas function access to S3 via ServiceAccounts and IAM

[shyamammu123]

We're trying to grant a specific function access to S3 via ServiceAccounts and IAM. However according to this bug https://github.com/kubernetes-sigs/external-dns/pull/1185 we need to either let the function run as root (not an option) or add to the security context of the function as a workaround. Such as : securityContext: fsGroup: 65534

We didn't find a way to do this with openfaas yet. We looked updating the faas-netes helm charts with the context but didn't have success there. Maybe someone can point us to the right file? (edited) 13:50

1185 Fix AWS IAM Roles for Service Accounts permission problem

Expected Behaviour

The openfaas function should be able to access the token present in the root filesystem.

Current Behaviour

2020/03/12 16:08:28 stdout: PermissionError: [Errno 13] Permission denied: '/var/run/secrets/eks.amazonaws.com/serviceaccount/token'

Possible Solution

Steps to Reproduce (for bugs)

1.Create an EKS cluster 2.deploy openfaas 3.grant s3 access to a service account via IAM roles (https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html) 4.create a function that accesses S3 with boto and executes as non root user

5. annotate the function.yaml with the service account such as : annotations: com.openfaas.serviceaccount: service-account-name

Context

Your Environment

• FaaS-CLI version ( Full output from: faas-cli version ): v0.11.6

• Docker version docker version (e.g. Docker 17.0.05 ): 19.03.1

• What version and distriubtion of Kubernetes are you using? kubectl version v1.16.1

• Operating System and version (e.g. Linux, Windows, MacOS): Linux

• Link to your project or a code example to reproduce issue:

• What network driver are you using and what CIDR? i.e. Weave net / Flannel AWS VPC

[alexellis]

@shyamammu123 thanks for raising the issue, please fill out the whole template including the steps to reproduce, and give granular steps i.e. assume no prior knowledge of your company and how you configure AWS.

Could you also give details about the custom patch @LucasRoesler created for you and what happened when you tried that?

Thanks, have a good weekend.

[LucasRoesler]

@alexellis i have update my commit so that it now references this issue

The test image is available at docker pull theaxer/faas-netes:fsgroup-poc

[jeremy271828]

The custom patch @LucasRoesler created does work to address the issue. The securityContext with the fsGroup allowed a non root user to access IAM credentials from within the functions created by faas-netes in an EKS cluster

Will this be merged to master? Thanks!

[shyamammu123]

@alexellis I have updated the issue with steps to reproduce and the environment details. Let me know if that information suffices?

[jeremy271828]

Will this also be added to openfaas-operator?

[LucasRoesler]

@jeremy271828 last time we talked on slack i thought it had no effect, are you sure it worked?

[LucasRoesler]

if this is definitely working, then we need to decide if this can just always be set every time we set a NonRoot user or if it needs to be configurable. It isn't clear to me we need to add another flag, it seems harmless to always set it, but i am no expert on linux permissions etc.

Does this work for windows containers too?

[jeremy271828]

Not sure about windows containers, but with the new patched images a pod describe shows: securityContext: fsGroup: 12000 And the non root user is able to access the IAM token file and access S3 buckets from the function.

It might be nice to be able to add user configuration settings to function deployments, but not a requirement, as long as the setting is there for non root users it works.

[LucasRoesler]

My concern is more about the unintended consequences of setting this value, is it possible to break some existing functions that expect this value to be different

[jeremy271828]

Right... it is safer to make it configurable.

[alexellis]

Just to check with you @jeremy271828 this would probably come into faas-netes first, then the operator at some point in the future. What's the reason for you deciding to use the operator in this instance?

[LucasRoesler]

@alexellis i was thinking more about this, I wonder if it makes sense to actually specify this via the secrets list. something like _serviceaccounttoken, we could then make sure that the file system if readable so that the service account token is available to the function.

The use case here very much is "i want access to a secret file". Perhaps there is a better name for this keyword or we can require that the name matches a service account name in the system. But this seems like a good way for the function itself to specify that it wants this access. It should be easy to audit and could be short extension of the current secrets system.

[aleks-fofanov]

@LucasRoesler @alexellis I also want this feature implemented. Actually this is the only reason I had to install kube2iam in EKS cluster in order to use IAM roles with OpenFaaS functions. In the long term I think it would be great to have an ability to pass down pod and container security contexts to deployment to harden security of their deployments, especially for those users who have PSPs in their cluster.

Here's the proposal. Let's add an ability define pod and security contexts for function deployments in function spec passed to the REST API and CRD (for those who deploy through operator). Existing users will be unaffected this change because be default both will be empty. The only downside is that we will have to deprecate readonly_root_filesystem in favor of security contexts.

How are both looking at this? If this looks good to you, then I would like to volunteer and implement this feature in my spare time.

P.S. Looks like other are interested in this too #660

[alexellis]

We are much more likely to implement this at the OpenFaaS deployment level, rather than at the Function schema. The various options include a mode and using the new profiles feature.

We need more context though. Do you see this configuration being applied to all functions in your cluster in the same way? Perhaps you can reply with some concrete examples

[spaghettifunk]

If I can add my 2 cents on this issue. We also would like to access aws resources from our openfaas functions, however, I do believe that define IAM Roles at Function level is better. Not all our functions need to access the same resource and I think we should avoid to have one unique role at OpenFaaS deployment level. This will allow folks to create specific roles with small policies 😄

[alexellis]

Thanks for your thoughts, however we need some concrete examples now, real-life needs from users to proceed.

[spaghettifunk]

You are right @alexellis . I can give you the few use cases we have at the company that I work for:

1. We trigger a function with the cron-connector that reads from S3 and push some logs to Elasticsearch
2. We trigger a function that needs to read from an S3 bucket to update our Redis instance

We do other 3 or 4 functions that do the above but for different S3 buckets (they are from different external parties). The role would help us to avoid adding the key and the secret as ENV variable (which I am doing right now as workaround 😄 ). Does this help?

[LucasRoesler]

Not sure about windows containers, but with the new patched images a pod describe shows: securityContext: fsGroup: 12000 And the non root user is able to access the IAM token file and access S3 buckets from the function.

It might be nice to be able to add user configuration settings to function deployments, but not a requirement, as long as the setting is there for non root users it works.

This setting should now be accessible via the Profiles feature

https://github.com/openfaas/faas-netes/blob/8a9c15aa9ecc81245e87837dec097a6fc8053c9b/pkg/apis/openfaas/v1/types.go#L113

You can set the entire PodSecurityContext object now: install this into your cluster

kind: Profile
apiVersion: openfaas.com/v1
metadata:
    name: withIAMTokenAccess
    namespace: openfaas
spec:
    podSecurityContext:
        fsGroup: 12000

and then add this annotation to your function

annotations:
      com.openfaas.profile: withIAMTokenAccess

[spaghettifunk]

Thanks @LucasRoesler! I'll give it a try tomorrow and I update the issue if it works 😄

[alexellis]

@spaghettifunk how did you get on with this?

[alexellis]

Closing as inactive. Feel free to raise a new issue if people still need this.