search

wandb / server

W&B Server is the self hosted version of Weights & Biases
MIT License
263 stars 21 forks source link

Really high memory usage due to mysqld #48

Open MattAlexMiracle opened 2 years ago

MattAlexMiracle commented 2 years ago

Hi, As of recent I've had stability problems with the wandb container. I narrowed it down to absurd memory usage of the mysqld: On my system (Ryzen 1600, 32GB ram, Arch-linux) a bare wandb instance spawns dozens of mysql processes that in total take up 16GB+ or memory. I was able to narrow it down to https://github.com/docker-library/mysql/issues/579 In mysql. I was able to confirm that the fix proposed there (running the container with --ulimit nofile=262144:262144 fixed the issue entirely (I still have a lot of mysqld processes, but they only take up a couple of mb in total).

Maybe consider adding this by default or document this as a known issue of mysqld so other people can find it?

vanpelt commented 2 years ago

Thanks @MattAlexMiracle this is really helpful. We'll look into adding this to default local command.