search

openfaas / faas-cli

Official CLI for OpenFaaS
https://www.openfaas.com/
Other
794 stars 227 forks source link

Security/Compliance - Validate function base image signatures #608

Open prahaladdarkin opened 5 years ago

prahaladdarkin commented 5 years ago

Currently, the base images used for creating and deploying functions are not validated for trust-worthiness during the function build process. However, enterprises are very security and compliance centric and hence must use images signed by well known/ trusted sources as the base image for their functions. This trust validation mechanism can be invoked during the function build process so that if the images are not trusted then the build fails and would not generate a function image. This is also an essential compliance requirement for all organizations that deploy code in production environments.

Expected Behaviour

faas-cli build should have a mechanism to validate the trust of base images used for creating functions by validating that the signers of the image are in the list of trusted users for an organization or department. If the image being used has no trust signatures or if the list of the signers of the image are not in the set of known trusted users; then the build should fail.

Current Behaviour

Currently, there is no trust or signature validation of base images used in functions. Hence, a function can be built using any base image. By implementing the proposed enhancement; the build will fail for un-trusted images or when the signers of the image do not come from a defined whitelist.

Possible Solution

The solution to implement the proposed change is described below:

openfaas_cli_trust_verification_algo

Context

By providing trust validation when building function images; we go one step forward towards adding the required checks in OpenFaaS to make it enterprise ready. There definitely exist ways to validate trust images offline and then provide these trusted images list to be used by developers. However, it still does not solve the problem of ensuring use of trusted images; since the function developer has to be trusted to use the correct images. By making trust verification a part of the function build process; these checks are enforced.

Your Environment

Server: Docker Engine - Community Engine: Version: 18.09.1 API version: 1.39 (minimum version 1.12) Go version: go1.10.6 Git commit: 4c52b90 Built: Wed Jan 9 19:41:49 2019 OS/Arch: linux/amd64 Experimental: true

prahaladdarkin commented 5 years ago

@alexellis I have already implemented the above algorithm as a PoC. This is the first time I am writing Go code; hence there would be some gaps. I have to write some additional tests post which I can send the code across for review.

alexellis commented 5 years ago

This looks like a very interesting topic. I just wanted to ask if you could cover some addition points before we spend time looking into it?

Have you also considered the functionality that is already provided by registries like Harbor and Quay? Most container pipelines work through a number of stages forming a pipeline with the container scanning being one of those steps or a policy in the cluster preventing the final deployment of images which are untrusted or which have vulnerabilities.

alexellis commented 5 years ago

If you have the code in a public GitHub repo I would suggest pasting a link here and recording a quick video for YouTube using QuickTime on your computer. If you can keep it under 5-10 mins that'd be great.

Alex

zeerorg commented 5 years ago

I think we need a bit more insights as to what approach organizations take for securing their environment, maybe different orgs take a very different approach than this, any links to official blog posts, or talks on this would be greatly helpful.

Openfaas Cloud secures its environment by only allowing functions that derive from the trusted template store, maybe this approach (trusted template store) might be better.

Also, just checking signature of the base image might/might not be sufficient. We should probably provide a way to sign images before being pushed and then checking them before they are deployed to have a more end to end secured pipeline.

prahaladdarkin commented 5 years ago

@zeerorg and @alexellis There exist articles or blog posts about Harbor such as the one from VMware here : https://blogs.vmware.com/cloudnative/2018/06/14/harbor-delivers-a-trusted-cloud-native-registry/ For quay.io here is a blog note I found which says they are still evaluating the docker trust model : https://support.coreos.com/hc/en-us/articles/115000289114-Docker-Content-Trust-support. There is no further update on this thread. All major cloud vendors - IBM, Azure have support signing images for trusted content. However, these rely on Docker Content Trust to validate content trust. For example here is a IBM Cloud documentation link for trusted content : https://console.bluemix.net/docs/services/Registry/registry_trusted_content.html#registry_trustedcontent It should be noted that DCT is not a fool-proof solution since as stated in the above blog; DCT is based on the Trust On First Use Model - which means that the user trusts the image for the very first time it is being downloaded and that docker client would validate trusts and signatures only after the first use. In general, content trust is available to users who use one of these services (Harbor/IBM Container Registry/Azure Container registry). But then there are companies out there who would use Docker Hub as the source of all their images and they also would want to perform trust verification. Also, to the best of my knowledge of working with OpenFaaS; when building a function, OpenFaaS has no knowledge of where the base image came from (@alexellis or others can correct me if wrong) With all these said, it would make a good case of having a command line option to double check the trust or signature of the image without making any presumptions about registry from where this image was fetched in the first place. With respect to the point of providing a way to sign an image - yes, that is definitely something that should be done. We should sign the final generated function image using the keys of whitelist users. I can take that as another enhancement PR.

prahaladdarkin commented 5 years ago

@alexellis @zeerorg Let me know if there are other comments or even if these changes are worth pursuing. I have other changes in the same area that I am going to propose soon for OpenFaaS and would raise them as separate Issues.

zeerorg commented 5 years ago

Great insight, you should also take into account alex's comments about providing a github repo link and a recorded video.

prahaladdarkin commented 5 years ago

I would be posting a video this weekend. I am swamped at work this week. Also, would send across a PR post uploading the video. A couple more enhancements that I am thinking of - If instead of having a file signers.txt; we could use Harbor or Quay API to fetch the authorized list of signers (possibly the project members) then we would be able to use that for validating the signature during builds. @alexellis I would work on this post the video submission

prahaladdarkin commented 5 years ago

@alexellis @zeerorg Here is a video that demonstrates base image signature validation during faas-cli build process : https://youtu.be/U8UL2lrDqlc

LucasRoesler commented 5 years ago

@prahaladdarkin i just watched the video and it is very well presented. I also like the idea that you present.

The current proposal looks like it would work well as a first step to the complete solution. The finally solution should probably be paired with a server side implementation. But this is a very good first step and would definitely give developers more peace of mind that the functions that they are building are 'safe'.

I like the idea of not needing to install or manage another component like Harbor or Quay in order to improve the build safety. Also this implementation does not preclude integrating with any other solutions; so, it can easily be paired with other solutions to build up a more complete compliance process.

Do you agree that the next steps would be :

1) support signing the function image, 2) verify signatures in the server?

For the server side, could we leverage Harbor and Quay by requiring that all function images come from specific trusted registries that already support the signature verification? This could be added to the openfass-provider as a simple string validation on the deploy request.

prahaladdarkin commented 5 years ago

@LucasRoesler thank you for the feedback :). Yes I agree that the next steps would be to support signing the function image and also verification of signatures at runtime. The original pipeline that had been conceived is attached : OpenFaas-Compliance-Validation-Proposal.pdf My colleague Michael Gasch also had similar idea/proposal of developing a custom web hook for validating image signatures and runtime. I would collaborate with him when I come to this phase of the implementation to deliver this functionality

alexellis commented 5 years ago

I have concerns on using a flat-file on disk to verify a list of names (as an ACL)

How does GPG do this with apt in Debian or pacman in Arch Linux? I believe public keys are brought into a keychain which exists locally and then the signed packages are verified that way.

Checking a name as a string seems to be lacking somewhat.

It would be good to have input from @justincormack or @stevvooe

prahaladdarkin commented 5 years ago

@alexellis I checked how apt in Debian works - they do bring-in the public keys in a keyring or also allowed to be added by the user and then the keyring is used to verify the package signatures. I am not sure if there is similar functionality available via docker command line. DCT uses manifests to validate that image signatures are valid. I agree with you that providing a simple text file to list allowed signers is unsettling. I did this only as a very initial step. As I have mentioned in my video; we could pull the list of users from other places too such as AD user groups, or project members in Harbor or Quay projects. In the absence of an AD/Harbor/Quay scenario; one way to get the list of users securely would be to create an OS group say "OpenFaaS_Signers" and add users to this group. This step would at least make tampering the whitelist of users more difficult; since it would require users to be added to the OS group and such activity can be audited. It would really however be good if docker provides an out of the box mechanism to list all trusted users for an repository. I have tried the existing docker manifest inspect command (experimental mode) and this does not give me the list of signers configured for the image. Similarly with the docker trust command set. Would I have ping @justincormack or @stevvooe on a separate channel?

alexellis commented 5 years ago

I would like some input from either of those two chaps, but I'm sure they are very very busy with their own stuff. They are both experts on Notary and Docker image signing / security.

This is interesting and looks useful, but I'm not sure what we'd gain by merging it in its present state.

Let's come up with a plan on how end-users could make practical use of these changes. I'll also ping @ewilde who may have some ideas.

Alex

prahaladdarkin commented 5 years ago

@alexellis @ewilde @LucasRoesler I am currently putting this PR on hold until we hammer out some of the specifics. I will in parallel continue contributing to other PRs and some parallel notary research on my own. @ewilde Request your inputs towards this PR - this will help in prioritizing this change (if at all required and relevant)

prahaladdarkin commented 5 years ago

@alexellis @LucasRoesler @ewilde Please refer to the below discussion with respect to DCT support in Kubernetes. https://github.com/kubernetes/kubernetes/issues/30603. The question here is in line with this proposal.

The tone of the discussion again points to a point that has been mentioned above - DCT is not a fool-proof mechanism. Also, the support for DCT in Kubernetes is hazy. Given the platforms that OpenFaaS can run on (and Kubernetes being an important one); there needs to be custom work put in to ensure signature validation when running function images. Similar to this proposal, the There are two questions that this proposal is trying to address:

  1. Are the images signed?
  2. If they are signed, are they signed by the roles that I actually trust?

DCT in some ways can help with question 1 but I am unable to find a reliable way in DCT to answer question 2. One of the approaches suggested in the above link was to have the CI write out signatures to a file and then some custom logic validates these signatures in the image layers. This is similar to proposal here where the user has to finally specify a whitelist of signers against whom the set of image signers is validated. The whitelist could be specified using various mechanisms; to start with it is a file based approach; but the design should be generic to use any other provider of whitelists such as AD groups or project member lists.