[Feature] Build-time secrets for private npm/nuget/artifact repositories

[chrisrichard]

Our FaaS functions reference NPM packages from our private NPM repository (via package.json) so when OpenFaaS Cloud does its npm install, these dependencies fail to install.

Expected Behaviour

npm install will pay attention to a .npmrc file if present, but the auth token itself needs to be provided as a secret.

Example from our Travis build:

echo @ratehub:registry=https://packagecloud.io/ratehub/npm/npm/ > .npmrc
echo //packagecloud.io/ratehub/npm/npm/:_authToken="\${NPM_TOKEN}" >> .npmrc

Example from Docker build:

ARG NPM_TOKEN
...
RUN    echo "@ratehub:registry=https://packagecloud.io/ratehub/npm/npm/" > .npmrc && \
       echo "//packagecloud.io/ratehub/npm/npm/:_authToken=$NPM_TOKEN" >> .npmrc

Current Behaviour

Doesn't seem to be a way to generate this config at the appropriate time and plumb secrets into a it. We could probably commit the .npmrc file to the Git repo if there was a way to have OpenFaaS Cloud substitute the secret.

Context

We have a lot of functionality already built into private NPM packages; in a lot of cases our FaaS functions are just stub/wrappers around this functionality.

[LucasRoesler]

This has also been discussed in some of the other big cloud tools out there, we should take a look at some of their research as well

• https://github.com/Azure/draft/issues/564
• https://github.com/moby/moby/pull/36443#issuecomment-369845823

Docker/Buildkit has some support for supplying secrets to the build context, but it is still marked as experimental : https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information

I have used this in some of my projects (it has some interesting build cache features too) but not the secrets flags. I am not sure if we can transparently turn this on for users, the Dockerfile needs to be modified to start with

# syntax = docker/dockerfile:1.0-experimental

If we could transparently turn this on with buildkit, then we could consider expanding the stack yaml spec to include build flags, one of which could be a list of secrets to include.

[alexellis]

Thanks for the input on this.

How does a user supply these secrets to their OFC or the community cluster when everything is done via git?

How do we pass the secret to the builder image when Kubernetes has to restart Pods to attach volumes and secrets are attached via volumes?

[alexellis]

I think that we are looking at something a bit like this.

[alexellis]

Whilst this doesn't help for the managed platform, some users are using build-args with the faas-cli to specify secrets for private npm.

[LucasRoesler]

It seems reasonable to reuse the existing secrets workflow, if the sealed secrets are deployed prior to building, then they would be available to the build step. Perhaps it makes sense to allow labels for secrets to restrict if a deployed function is allowed to use a build secret

[alexellis]

OFC uses buildkit as a daemon at the moment, which may present some problems with that approach. If one shot containers are created for each build then the approach may work.

[alexellis]

How are you using buildkit at the moment? One shot or as a daemon?

[LucasRoesler]

We use a daemon in our Jenkins CI, but we don't use the secrets support, only the RUN cache https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md#run---mounttypecache (this makes building go projects great, by the way)

I am not sure that the buildkit daemon needs to mount the secret, I think that only the client that builds and sends the build context needs access. See the example in buildkit https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md#run---mounttypesecret The secret file just becomes part of the build context and is mounted in a special way

[csakshaug]

We have similar issues with private PHP composer repositories and could perhaps have the same problem with private GO imports/modules and private nuget repositories.

[lskillen]

This would be very worthwhile. If you need a testing plane for this, happy to provide a free @cloudsmith-io account to try private authentication against all of the formats we support.

[alexellis]

Thanks @lskillen are you also on OF Slack?

[alexellis]

/msg: slack

[derek[bot]]

-- Join Slack to connect with the community https://docs.openfaas.com/community

[lskillen]

@alexellis I'm not (yet), but I'll apply to join. :-)

[alexellis]

@csakshaug FYI any solution here would be generic for build-time secrets, not just for npm, npm is just the use-case that was proposed first.

cc @AkihiroSuda

[alexellis]

Some limitations on --build-arg at present -> https://github.com/moby/moby/pull/36443#issuecomment-369843123

[alexellis]

There are two options I'm exploring:

A) Deploy a new function in OpenFaaS for every build which has a secrets: section and binds each of the build secrets the user wants. It will invoke of-builder instead buildshiprun, afterwards that function will be deleted. This option will introduce additional latency which is not currently present since schedule a Pod in Kubernetes can take 2-5 seconds+/-.

B) Grant the buildshiprun function RBAC access to all secrets for all users (if possible, restricted to match only those that we know are build-secrets). It will ask the Kubernetes API for the plaintext value for each secret and then build a HTTP request to send to of-builder.

For importing secrets we have a mechanism already with SealedSecrets.

Both A and B look better if we implement namespace segmentation first in #533

[alexellis]

A temporary workaround which I've not shared yet is to build sensitive images in a separate CI/CD workflow, and then have OpenFaaS Cloud pull them in using the dockerfile template.

An example exists here for kubesec

https://github.com/teamserverless/kubesec/blob/master/kubesec/Dockerfile#L1

The last step of your Travis/Jenkins pipeline would be to update the Dockerfile in the repo linked to OFC.

[csakshaug]

A temporary workaround which I've not shared yet is to build sensitive images in a separate CI/CD workflow, and then have OpenFaaS Cloud pull them in using the dockerfile template.

An example exists here for kubesec

https://github.com/teamserverless/kubesec/blob/master/kubesec/Dockerfile#L1

The last step of your Travis/Jenkins pipeline would be to update the Dockerfile in the repo linked to OFC.

I am doing that now, it works

[Waterdrips]

There are two options I'm exploring:

A) Create a new function which has a secrets: section and binds each of the build secrets the user wants. It will invoke of-builder instead buildshpirun, afterwards that function will be deleted. This option will introduce additional latency which is not currently present since schedule a Pod in Kubernetes can take 2-5 seconds+/-.

B) Grant the buildshiprun function RBAC access to all secrets for all users (if possible, restricted to match only those that we know are build-secrets). It will ask the Kubernetes API for the plaintext value for each secret and then build a HTTP request to send to of-builder.

For importing secrets we have a mechanism already with SealedSecrets.

Both A and B look better if we implement namespace segmentation first in #533

With B we are sending the secrets file over the network to the builder via http? I'd prefer not to expose build secrets in this way.

Does A involve building a new container, to mount the specific build secrets, to then use those in building the user's functions? While this adds some latency, it would expose less to the rest of the cluster.

[alexellis]

Both options send the secret over the network. If this is a concern people can install Linkerd for mutual TLS.

[frossi85]

Hi guys, I am wondering if we have any progress or estimate on this? Thanks

[alexellis]

If there's some customer interest in the feature then we may prioritize it. Tell us more about your usecase?

[frossi85]

Basically I have a Kubernetes cluster with OpenFaas Cloud installed using Github Auth and the default GitHub integration that runs an action to test/deploy your function to Cloud. I am using a custom Typescript function, that I hope to release to the public soon.

I found myself in the need to have a private npm repo to share some business logic across different functions, like auth stuff and other things related to the business logic. Because this is a university project, I try to keep the budget low and learn as much as I can. So I found https://verdaccio.org/ which is an opensource project to have your own npm registry with fallback in npmjs, and would like to use it during build time to retrieve dependencies without exposing the auth token in my template code.

I would not like to have to use another tool for CI, I already dedicated too much time to verdaccio and I am behind schedule. So I would to keeping Github integration if possible. Maybe write some custom github action with secrets? Do you use that inside the Github integration on openfaas cloud?

[frossi85]

But besides my use case, I know is pretty common for people that are not doing open source to have their own private repositories for things like npm and maven as some of the examples. So I think this feature will help a lot of people and make openfaas cloud a more enterprise-ready solution.