search

ArmyCyberInstitute / cmgr

CTF Challenge Manager
Apache License 2.0
17 stars 9 forks source link

Keyring quota exhaustion #40

Closed dmartin closed 2 years ago

dmartin commented 2 years ago

Not a cmgr bug, but a potential gotcha that might be good to document, similar to #39.

I've found that it's pretty easy to run into the default keyring quotas when running lots (~200) of containers.

The error bubbles up from runc as:

error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: join session keyring: create session key: disk quota exceeded: unknown

when creating a build.

This blog post mentions a similar situation, though in their case they were able to work around it by increasing only /proc/sys/kernel/keys/maxbytes. I found that I needed to increase /proc/sys/kernel/keys/maxkeys as well, though mine was originally set to the Ubuntu default 200 rather than the 1000000 mentioned in the blog post.

dmartin commented 2 years ago

As a clarification, I think that the non-root keyring limits only come into play when user namespace remapping is enabled for the Docker daemon. Since that's a non-default configuration it's probably not appropriate for the cmgr docs to address it.