Helm chart for Notary

[patoarvizu]

I started exploring Notary a couple of days ago and I personally find it easier to run things locally on Kubernetes (with minikube or k3s) rather than docker-compose, but I couldn't find a Helm chart for Notary anywhere so I started writing one.

I plan on submitting a PR with a more polished chart in a couple of days, I just wanted to open the issue first to get visibility.

[HuKeping]

Do we really want introduce helm to this git repo(for notary source code).

Actually I don't like the docker-compose.yml to be here either, but since it could be a guide to show people how to deploy notary-server and notary-signer and is simple enough, so I haven't drop it.

I'm thinking if we should create another separated git repo to hold those templates-such as notary-template-helm, notary-template-docker-compose rather than tie it up with this git repo.

thoughts @theupdateframework/notary-maintainers ?

[patoarvizu]

One of the benefits I can see from splitting things into different repos is that we would be relying less on shared things like manifests, config files with very specific options, documentation, etc. that needs to work for both. If we go that way, we could even abstract things out a little more and instead of providing these things, add automation on how to create them (go templates, openssl, or step commands, or Terraform running locally with the TLS provider, etc. This way, any changes on one solution won't risk breaking the other (and requiring more specific knowledge to fix them).

On the other hand, I think part of the value of adding this here, is that it shows activity and signals interest in the project (in the context of this proposal). I feel that keeping things separate will decrease visibility into the activity of the project. Furthermore, having the examples co-exist with the core code can (in my opinion) remove the cognitive wall of having two separate repos, and potentially encourage more community contributions. In other words, I think the docker-compose and Helm examples can be a form of "gateway code" 🙂

[HuKeping]

Furthermore, having the examples co-exist with the core code can (in my opinion) remove the cognitive wall of having two separate repos, and potentially encourage more community contributions.

I agree it may increase the activity of this project if we put these two together, but in the meantime I also think it doesn't make much sense if the target is to bring more contributor to this community.

I've noticed the proposal at CNCF TOC, I think they're just doing their job, supervise the projects under CNCF and to see if there's any thing that they could help.

[HuKeping]

Furthermore, having the examples co-exist with the core code can (in my opinion) remove the cognitive wall of having two separate repos, and potentially encourage more community contributions.

I agree it may increase the activity of this project if we put these two together, but in the meantime I also think it doesn't make much sense if the target is to bring more contributor to this community.

On the other hand, I think part of the value of adding this here, is that it shows activity and signals interest in the project (in the context of this proposal).

I've noticed the proposal at CNCF TOC, I think they're just doing their job, supervise the projects under CNCF and to see if there's any thing that they could help.

[justincormack]

I am not particularly keen on having a chart that is not suitable for production in this repo. It could go elsewhere, but it would really need ongoing CI as well.

[patoarvizu]

@HuKeping It's not clear to me if you're saying that you don't think it makes sense to bring more contributors to the community, or if you're just not sure if that's the target.

@justincormack Would you prefer if this was polished to make it production-ready? Or put it in a separate repo while it gets polished and eventually move it here? I can do either one.

[justincormack]

I think it is better to make it production ready.

[patoarvizu]

Definitely. I'll work on it and ping here again when I think it's ready to be re-reviewed.

[HuKeping]

@HuKeping It's not clear to me if you're saying that you don't think it makes sense to bring more contributors to the community, or if you're just not sure if that's the target.

Sorry for the misleading. I mean if the reason we bring in Helm chat is to bring more contributors to the community or to increase some index about activity of this community, that would not make much sense.

[patoarvizu]

@justincormack I've reworked the chart quite a bit and made it much more production-ready.

Here's a list of the things I added:

• No more pre-made certificates. The chart allows you to inject your own or it can auto-generate its own certs, with some parameters.
• The values.yaml file has comments for every option.
• The chart now can use Kubernetes Secrets to avoid rendering them in clear text. Previously, it was generating ConfigMaps, or commands or environment variables in Deployments that contained secrets in plain text, but now that's abstracted out. The only exception to rendering secrets is when a password is explicitly passed as a value, instead of referencing a secret.
• Added affinity options, now the pods can be configured to not be colocated or to prefer scheduling on a specific set of nodes.
• The storage pods are now optional, and Notary can be configured to use remote storage.
• Added authentication and caching options.
• The migrate operations are now done in one-off jobs, instead of initContainers.
• Removed symlink to notarysql and fixtures, now it's only symlinked to migrations.

Is there anything else you would like to see?

Any thoughts on how to CI this? If running on CircleCI, we can easily use k3d to quickly spin up k3s clusters, but I'm not sure how we can API-test a running Notary service. Is there anything in the codebase that does that already? Otherwise, I guess we can at least use kubeval to validate that valid manifests are being rendered.

[patoarvizu]

Has any of you had any opportunity to take another look? If you don't think this belongs on this repo or have no plans on merging it, I would love to publish it myself.

By the way, as an alternative, I also wrote a Terraform module to deploy Notary and published it in the Terraform registry (https://registry.terraform.io/modules/patoarvizu/notary/kubernetes), if you're interested. It's a rough version, but the idea is pretty much the same, to provide a way to quickly spin up a Notary deployment and has pretty much the same feature set as the Helm chart.

Thanks!

[perezjasonr]

Any updates on this? I think having a helm chart for notary would be great, im actually surprised it didnt already exist, which is how i found this open issue.

[patoarvizu]

@perezjasonr I've kinda given up on following up on this. I'm not sure if the maintainers are uninterested or indifferent, but I haven't heard anything back from them in several months. I don't know how close they are to finalizing Notary v2. It's possible that at this point it doesn't make sense to merge this, but it still is quite disappointing that I was initially encouraged to contribute only to get no attention later.

FWIW, this chart was written with Helm2 in mind, which is EOL at this point. If there's enough interest, I can try and rework what's necessary, make it work with Helm3 and put this in a separate repo.

[perezjasonr]

@patoarvizu i think this is a project worth pursuing and getting it going with helm3. even if notary v2 is coming down the road can't the helm chart be updated to work with notary v2? could even be a config entry saying which notary you want etc...

what skills are needed to contribute to this? I may even get on board myself.

[patoarvizu]

I think Notary v2 is not going to be just an image version change, it's a re-architecture of the system, so the chart will probably need to be entirely different.

If you want to take a stab at it, you can probably take what's on this branch (or fork my fork) and move it to a new repo and start there. Pretty much everything I made here can be tested on a local Kubernetes cluster (with k3d, kind, docker-for-desktop, etc.), so you shouldn't need external infrastructure.

I may take it on as a holiday project soon, but if you can tackle it first, feel free!

[perezjasonr]

Ok I didnt know the difference would be that extreme...ill have to think about it if thats the case.

[developer-guy]

here is a useful one: https://github.com/k8s-gadgets/k8s-content-trust from medium post

https://siegert-maximilian.medium.com/ensure-content-trust-on-kubernetes-using-notary-and-open-policy-agent-485ab3a9423c