search

strangelove-ventures / cosmos-operator

Cosmos Operator is a kubernetes operator for managing cosmos nodes
Apache License 2.0
82 stars 20 forks source link

infra-toolkit busybox tar causes k8s node OOM #436

Open bizrad opened 1 month ago

bizrad commented 1 month ago

We are using the cosmos operator in our k8s cluster. When downloading the snapshot in the init container that runs ghcr.io/strangelove-ventures/infra-toolkit:v0.1.6 our k8s node with 256GB of ram gets OOM killed. Before this happens this can be observer by watching free -h under "buff/cache". This is memory used by the kernel to cache disk I/O operations and therefore not included in pod memory limits or usage tracking.

The infra-toolkit image is based on busybox and uses a stripped down version of tar. Our snapshot is compressed with lz4 and is being extracted by this script in the init container.

I was able to run a test and simply switching to an Ubuntu image with GNU tar in place and running the same commands resolved this issue. GNU tar is better suited for handling large archives with optimized memory usage.

Would it be possible to switch the image to GNU tar and put this in the path before busybox tar?

Alternatively, is it possible to swap out the init container image?

agouin commented 1 month ago

Thank you for the issue! We also use lz4 compressed snapshots, so I am surprised that we have not run into this ever for our deployments. We will review the switch to gnu tar in infra-toolkit and publish a new operator version if accepted.

is it possible to swap out the init container image?

Yes, this is possible with the strategic merge patching of the spec.podTemplate, e.g.:

apiVersion: cosmos.strange.love/v1
kind: CosmosFullNode
spec:
  podTemplate:
    initContainers:
       - name: snapshot-restore
         image: $CUSTOM_IMAGE_REPO:$CUSTOM_IMAGE_TAG
bizrad commented 1 month ago

It's possible the reason could be differences in resources. I'm seeing DL speed of about 800Mb/s. This was seen on a mostly unused node using 8 of 48 cores at idle. Since swapping out tar makes the difference I'm guessing it's most likely that the lz4 decompress is faster than busybox tar can handle in our environment.

nourspace commented 1 month ago

@pharr117 did we finalise the test in https://github.com/strangelove-ventures/infra/pull/3336?

nourspace commented 1 week ago

Ping @pharr117