harshavardhana
commented
2 years ago mc uses GOMAXPROCS to run in parallel so it can potentially use memory you need two things to set - change GOMAXPROCS to make sure it uses less concurrency.
With --watch the issue is that the memory that is used to upload is never GC'ed fast enough. So you hit the upper GC overhead as well.
So you need to reduce the concurrency such that memory is relinquished cleanly with GOGC=25 and then set GOMAXPROCS=4
However
- it would be nice to know how much memory is given to the
mcpod? - when it does OOM what is the total memory used?
- --watch do you have a lot of incoming objects on source? and what is the average object size?
chrisbecke
vadmeste
Vane-L
Expected behavior
The container version of minio/mc should use the available S3 APIs and mirror minio servers no matter their age.
Actual behavior
The containerized version of mc eventially consumes all memory and is OOM killed by docker.
Steps to reproduce the behavior
I am running an older minio server as a source server. It was installed circa 2020 and for regulatory reasons has not been updated. It is a simple minio in front of a single disk setup. The target minio instance is a 4 node cluster that is up-to-date.
I run mc as a docker container on docker swarm using this stack file:
iobconfig pointing to a new minio instance and azfpconfig pointing to an older - circa 2020 - minio instance. Have a "downloads" bucket on the old instance you want to mirror to the new instance.docker stack deploy --compose-file sync-stack.yml syncdocker service ps sync_sync-zfp-iobmc --version
System information
Additional Info
Without the addition of
--watchthe command performs the mirror operation pretty flawlesslessly, but we need to mirror new content as it arrives. At any rate, if this operation is not supported,mcshould fail outright, or log errors to the prometheus endpoint, NOT leak memory and die.