Memory leak on ZEO server

hvelarde commented 7 years ago

On some production sites we are using ZEO on a ZRS configuration and I recently started spotting very high memory consumption on the zeoserver process (as high as 1.6GB on one server I just had to restart).

We have currently 2 sites with this issue:

site C: 1 database; 17,277,281 objects, 2 ZEO servers (one ZRS master and one ZRS slave); each ZEO server has 3 ZEO clients attached to it
site R: 1 database; 18,460,812 objects, 2 ZEO servers (one ZRS master and one ZRS slave); each ZEO server has 2 ZEO clients attached to it

Memory usage increases gradually with every database pack in around 170MB steps; and the leak happens with and without ZRS.

Some graphs can be seen here: https://community.plone.org/t/very-high-memory-consumption-on-zeo-server-process/4208

Versions used:

plone.recipe.zeoserver = 1.2.9 ZopeUndo = 2.12.0 ZODB3 = 3.10.5 zope.mkzeoinstance = 3.9.5 zc.recipe.egg = 1.3.2 zc.buildout = 2.5.3 zc.zrs = 2.4.4 zope.interface = 3.6.7 zope.event = 3.5.2 zdaemon = 2.0.7 ZConfig = 2.9.3 zc.lockfile = 1.0.2 transaction = 1.1.1 Twisted = 16.6.0 incremental = 16.10.1 constantly = 15.1.0

azmeuk commented 6 years ago

Hi. Can you provide a minimalist project that reproduces the memory leak?

hvelarde commented 6 years ago

well, that's pretty hard; as I mentioned it happens in 2 different projects, but ZODB have more than 100.000 objects each. I have no idea how to handle that.

zopefoundation / ZEO

Memory leak on ZEO server #97