BPF debugger creates one file per stream containing its protocols and never deletes it. Streams in Mina are quite short-lived, and during a node's lifetime tons of streams are created. As a result, the directory is polluted with millions of files (on one of the nodes, it is 1047560 files after <3 days of uptime).
Also it looks like these files never closed (at least while connection is alive). This also causes virtiofsd to overrun its open files limit, and breaks the whole k8s node.
vlad9486
commented
1 year ago
BPF debugger creates one file per stream containing its protocols and never deletes it. Streams in Mina are quite short-lived, and during a node's lifetime tons of streams are created. As a result, the directory is polluted with millions of files (on one of the nodes, it is 1047560 files after <3 days of uptime).
Also it looks like these files never closed (at least while connection is alive). This also causes
virtiofsdto overrun its open files limit, and breaks the whole k8s node.This data should be stored in the DB.