additional ca-bundle target option: pvc

[lknite]

I have encountered the situation where the size of a ca-bundle exceeds the allowed size of a configmap and/or secret.

Options:

• the allowed size of a configmap and/or secret can be increased, though this will increase what's allowed cluster-wide
• use a pvc instead, which has no size limit defined, instead the size limit is handled through other means, often a csi provisioner

Reference:

• additional use cases exist on the source side: https://github.com/cert-manager/trust-manager/issues/379

[erikgb]

This use case is probably better supported by trust-manager csi-driver, @ThatsMrTalbot? 😉

[ThatsMrTalbot]

In its current implementation the POC CSI driver loads the bundle from the secret/configmap so would have the same issue.

However I would not want to implement syncing to a PVC as targeting a PVC brings in more complications, for example:

• ReadWriteOnce would require is to create a volume for every Pod, creating/syncing a volume on pod creation would block pod startup.
• Not all CSI drivers implement ReadWriteMany.
• Some CSI drivers create volumes that have zonal restrictions, meaning we would need to run trust manager in every zone.

Other implementation options:

• CSI driver (as mentioned above)
• Init container that writes the bundle to an emptyDir

[lknite]

• emptydir's are not always allowed cause they use space on a worknode hard drive, but generally pvcs are
• i don't know about zonal restrictions, but would you say that's so common as to really be an issue?
• folks choosing the pvc option are doing so because configmap and secret will not work due to size restrictions, so waiting a bit when needed may be a completely acceptable choice

[ThatsMrTalbot]

I have put a bunch of thought into the implementation details, and I really don't think writing to PVCs is a feasible option.

If the CSI only supports ReadWriteOnce (EBS for example):

• ReadWriteOnce cannot be attached to multiple Pods
• Since multiple Pods cannot mount the same volume, a PVC must be created per Pod.
• The moment we create the PVC, it will be attached to the target Pod meaning the trust-manager pod cannot mount it to write the bundle.
• We would have to create a different PVC, create a VolumeSnapshot then create the Pods PVC using that VolumeSnapshot
• VolumeSnapshots are not supported by every CSI provider.
• Even with all this, the Pod startup delay would be massive.

If the CSI supports ReadWriteMany (EFS for example):

• We can create a single PVC and mount it to multiple Pods.
• If the CSI mounts can only only be mounted in a single AZ, we would need to create a PVC per zone and run a trust-manager per zone in order to write the bundle to the PVC.

Using PVCs creates a massive dependency on a specific feature set that CSI must implement.

---

In regards to some of your comments:

emptydir's are not always allowed cause they use space on a worknode hard drive, but generally pvcs are

EmptyDir can be used to create a in-memory FS with a size limit, not writing anything to hosts disk. The limit would not even need to be that big, 5mb can hold a lot of certificates.

i don't know about zonal restrictions, but would you say that's so common as to really be an issue?

The AWS EBS CSI driver has zonal restrictions. The AWS EFS driver may have zonal limits, it depends on configuration.

---

Honestly I think the perfect solution is writing our own CSI driver, this would have the following benefits:

• No startup delay, the driver we write would handle the bundle aggregation and write it to a tmpfs mount.
• No size limit since the driver is doing the aggregation.
• No dependency on another CSI provider with a specific feature set to create PVCs

[arsenalzp]

Do you already have project for CSI driver for this purpose? I can try to contribute.

[erikgb]

Do you already have project for CSI driver for this purpose? I can try to contribute.

No, not yet. @ThatsMrTalbot has create a POC for it, but not yet anything official. :-)

[ThatsMrTalbot]

After a demo at the last community meeting the POC was moved into the cert-manager org: https://github.com/cert-manager/trust-manager-csi-driver

Currently it loads the secret/configmap that trust-manager created, but that can be changed so it can perform the aggregation itself and thus remove the 1mb limit.

This is still very much in the early stages, but if you want to contribute to the design/build then feel free to get involved, a good place to start is our daily stand-ups or bi-weekly community meetings. See https://cert-manager.io/docs/contributing.