Possible to use as Kubernetes KMS provider?

[uumo]

Hi! I was hoping to use sops as a KMS provider to encrypt secrets at rest in Kubernetes ( https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/ )

I run a sops keyservice

sops keyservice --network unix --address /root/sops/sops.sock

And configure Kubernetes to use it:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          name: myKmsPlugin
          endpoint: unix:///root/sops/sops.sock
          cachesize: 100
          timeout: 3s
      - identity: {}

But I get an error

kubectl get secret somesecret -o json | kubectl replace -f -
Error from server (InternalError): error when replacing "STDIN": Internal error occurred: failed get version from remote KMS provider: rpc error: code = Unimplemented desc = unknown service v1beta1.KeyManagementService

It seems the protocol buffers of sops / Kubernetes kms aren't compatible

Does anyone know any other projects, or workarounds to use sops as a Kubernetes KMS without going down the rabbit hole learning golang & protocol buffers?

[autrilla]

You could write a proxy (doesn't have to be in golang), but I'm not sure what the upside is over using a cloud provider's KMS implementation directly

[uumo]

Thanks, I'll look into writing a proxy

It's a bare metal cluster with a smart card, so no cloud provider kms is available

[mhumeSF]

You could also use something like https://github.com/kubernetes-sigs/aws-encryption-provider on bare metal. Got this working with k3s on a local machine.

Is the smartcard something like a yubikey with gpg keys?

[uumo]

I don't understand, when you setup k3s did you modify aws-encryption-provider to work without AWS? The readme says: "Assumptions: You have an AWS account"

I'm using a nitrokey that works well with gpg and sops, I just haven't found a way to use it for Kubernetes secrets

[mhumeSF]

I did not modify the kms plugin to work without aws. The aws-encryption-provider proxies requests to kms service that runs in aws.

[uumo]

@mhumeSF aws-encryption-provider won't work. I need to encrypt Kubernetes secrets using a physical HSM in a potentially offline cluster, so I can't use AWS or any other cloud based KMS

SOPS is very close to a working solution, but as I wrote before the protobuf implementation isn't compatible with Kubernetes'

[uumo]

I added a bounty to this issue! I'll accept any solution that allows Kubernetes secrets to be encrypted with SOPS as a backend. It must actually run and work so please make sure you're up for setting up Kubernetes to test it.

I was thinking something like: sops keyservice --network unix --kube-kms-address /root/sops/sops.sock ( which should be compatible with Kubernete's KMS gRPC api )

I'll accept a fork, not requiring this get merged to award bounty. If maintainers want to merge it yay :)

[Chocobo1]

I think I'm able to help with this, but I need some clarifications to clear things up.

I'm using a nitrokey that works well with gpg and sops

@uumo Can you elaborate how do sops interacts with your nitrokey? i.e. what arguments did you supply to sops?

It must actually run and work so please make sure you're up for setting up Kubernetes to test it.

I have a working prototype and my next step is trying it out with Kubernetes.

I'll accept any solution that allows Kubernetes secrets to be encrypted with SOPS as a backend.

Would you accept a standalone program instead? The program will sit between Kubernetes and sops, as such:

           --->           --->
Kubernetes       program        sops
           <---           <---

Sops already defined their own keyservice protocol format and I suppose the maintainers are unwilling to support another protocol format. IMO a standalone program would be better than a fork because a fork needs active maintenance (git rebase) to keep up with the latest upstream changes, while the keyservice protocol format is already well-defined and is unlikely to have big revision in the future.

Project is at: https://github.com/Chocobo1/ksops

[Frizlab]

@uumo I have a preliminary implementation here: https://github.com/Frizlab/sops/tree/dev.kms_keyservice

To test, clone my fork, checkout the dev.kms_keyservice branch, then do this:

go build go.mozilla.org/sops/v3/cmd/sops
sops keyservice-k8s --verbose --config /path/to/config.yaml --path /path/to/sops.sock

The --config flag is mandatory (at least for now).

Let me know how it works for you!

[Frizlab]

@uumo Can you please have a look? Before continuing working on this, I’d like to be sure what I did works for you. In theory your requirements are met. Thank you.

[uumo]

Hi @Frizlab! Thanks for your implementation & sorry for the delay getting back to you.

I was able to compile and run your code, but was confused by the mandatory --config flag and content of the file /path/to/config.yaml in your example

Also to be clear, I need this to work with a gpg managed key stored on a hsm smartcard. For example, let's say I'm using the key with fingerprint 6CB8BB1311E113F1 returned by:

gpg --card-status
General key info..: pub  rsa4096/6CB8BB1311E113F1 2021-05-18 uumoo

So just as I can encrypt a file with:

SOPS_PGP_FP="6CB8BB1311E113F1" sops -e README.md

I expect to be able to do the same with this implementation:

SOPS_PGP_FP="6CB8BB1311E113F1" ./sops keyservice-k8s --verbose --path sops.sock

However when I try to apply a Secret resource I get:

knuc apply -f testsecret.yaml
Error from server (InternalError): error when creating "testsecret.yaml": Internal error occurred: rpc error: code = InvalidArgument desc = cannot generate data key: [failed to encrypt new data key with master key "FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4": could not encrypt data key with PGP key: golang.org/x/crypto/openpgp error: key with fingerprint FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4 is not available in keyring and could not be retrieved from keyserver; GPG binary error: gpg binary failed with error: exit status 2, gpg: FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4: skipped: No public key
gpg: [stdin]: encryption failed: No public key
 failed to encrypt new data key with master key "D7229043384BCC60326C6FB9D8720D957C3D3074": could not encrypt data key with PGP key: golang.org/x/crypto/openpgp error: key with fingerprint D7229043384BCC60326C6FB9D8720D957C3D3074 is not available in keyring and could not be retrieved from keyserver; GPG binary error: gpg binary failed with error: exit status 2, gpg: D7229043384BCC60326C6FB9D8720D957C3D3074: skipped: No public key
gpg: [stdin]: encryption failed: No public key
]

I'm guessing this has to do with the mandatory --config flag?

[Frizlab]

Yeah I did not implement any other way than the --config flag to specify the keys with which to encrypt the data. That is actually what I want to do next if you can validate what I did works :-)

If I’m not mistaken, the SOPS_PGP_FP env var, is the same as specifying the -p argument, which is the same as specifying a --config file which contains

creation_rules:
  - pgp: 'THEKEYID'

[uumo]

@Frizlab your code works beautifully & fully satisfies my requirements!

I tried to award the bounty but get an error "We had an issue to make this transfer". I saw @alexanmtz in this thread, I think he's a gitpay maintainer, maybe he can help?

[alexanmtz]

Hey @uumo, I'm here, and I'm so happy that @Frizlab solved your issue. 🎉

@Frizlab, there's a new requirement to transfer with Stripe; for security reasons, we now need the phone number, so can you please send it to me alexandre@gitpay.me so I can complete the information needed to send the bounty to your bank account?

We're updating the platform to require this field, but it's not implemented yet.

[alexanmtz]

Hey @uumo, now the @Frizlab bank account is fully set, so can you try to pay the contributor again?

[isindir]

@uumo, may not answer your original question, but there are few controllers which manage secrets made in a form of sops encrypted files, I'm maintaining this one: https://github.com/isindir/sops-secrets-operator

[prein]

@isindir I couldn't find an end-to-end example for the operator you maintain. Does it use a kms-plugin under the hood?

[civts]

Hi all 👋

I created a project to address this concern too. You can find it at https://github.com/markhork8s/markhor. It's similar to the one created by @isindir, but with an added focus on protecting the integrity of the Secrets. Originally, I created it for myself, but I think others may benefit from it as well.

If you want, please take a look and provide any feedback you may have on how to make it better.