Memory leak in websockets' permessage_deflate.py when opening/closing browser

[Aron3d]

Hey everyone! First of all, thanks for creating this wonderful library!

I'm noticing a memory leak when opening/closing the browser many times in a python application. Am I using the library incorrectly, is there some connection that I should close manually?

Ive investigated whether the chrome processes are cleaned up correctly, but that does not seem to be the problem.

tested platforms:

OSX 10.13.5 Ubuntu 16.04

versions:

python 3.6.6 pyppeteer 0.0.25

import asyncio
import tracemalloc

from pyppeteer import launch

async def main():
    browser = await launch()
    await browser.close()

tracemalloc.start()

for i in range(200):
    asyncio.get_event_loop().run_until_complete(main())

    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('lineno')
    print('tracemalloc top 10:')
    for x in top_stats[:10]:
        print(f'tm: {x}', x)

I see the used memory climb ~300 kb every time the browser is opened and closed, apparently without bounds. Tracemalloc seems to be pointing to something in websockets.

output on OSX:

tracemalloc top 10:
tm: redacted/lib/python3.6/site-packages/websockets/extensions/permessage_deflate.py:457: size=51.2 MiB, count=1601, average=32.8 KiB redacted/lib/python3.6/site-packages/websockets/extensions/permessage_deflate.py:457: size=51.2 MiB, count=1601, average=32.8 KiB
tm: redacted/lib/python3.6/site-packages/websockets/extensions/permessage_deflate.py:518: size=6400 KiB, count=200, average=32.0 KiB redacted/lib/python3.6/site-packages/websockets/extensions/permessage_deflate.py:518: size=6400 KiB, count=200, average=32.0 KiB
tm: redacted/lib/python3.6/subprocess.py:693: size=3284 KiB, count=990, average=3397 B redacted/lib/python3.6/subprocess.py:693: size=3284 KiB, count=990, average=3397 B
tm: redacted/lib/python3.6/site-packages/websockets/extensions/permessage_deflate.py:453: size=1469 KiB, count=802, average=1876 B redacted/lib/python3.6/site-packages/websockets/extensions/permessage_deflate.py:453: size=1469 KiB, count=802, average=1876 B
tm: redacted/lib/python3.6/json/decoder.py:355: size=517 KiB, count=6725, average=79 B redacted/lib/python3.6/json/decoder.py:355: size=517 KiB, count=6725, average=79 B
tm: redacted/lib/python3.6/site-packages/websockets/http.py:294: size=337 KiB, count=5798, average=59 B redacted/lib/python3.6/site-packages/websockets/http.py:294: size=337 KiB, count=5798, average=59 B
tm: redacted/lib/python3.6/site-packages/pyee-5.0.0-py3.6.egg/pyee/__init__.py:150: size=281 KiB, count=3201, average=90 B redacted/lib/python3.6/site-packages/pyee-5.0.0-py3.6.egg/pyee/__init__.py:150: size=281 KiB, count=3201, average=90 B
tm: redacted/lib/python3.6/asyncio/base_events.py:276: size=198 KiB, count=1798, average=113 B redacted/lib/python3.6/asyncio/base_events.py:276: size=198 KiB, count=1798, average=113 B
tm: redacted/lib/python3.6/asyncio/base_events.py:285: size=179 KiB, count=1366, average=134 B redacted/lib/python3.6/asyncio/base_events.py:285: size=179 KiB, count=1366, average=134 B
tm: redacted/lib/python3.6/site-packages/pyee-5.0.0-py3.6.egg/pyee/__init__.py:81: size=153 KiB, count=1397, average=112 B redacted/lib/python3.6/site-packages/pyee-5.0.0-py3.6.egg/pyee/__init__.py:81: size=153 KiB, count=1397, average=112 B

Any help would be greatly appreciated!

[Edwardx89]

@Aron3d I've noticed this too recently. Setting autoClose in launch_args to False seems to help with some of the memory leaks.

[Aron3d]

@Edwardx89 thank you so much for the suggestion!! Setting autoClose to False indeed solved my issue. The memory consumption seems to stay within bounds and I don't run out of open file handles anymore.

Since this is a workaround and not an actual fix for the issue, I don't think it should be closed just yet? Anyways, lots of thanks!

[Edwardx89]

@Aron3d No problem!

@miyakogi Any idea why the memory leak is happening with autoClose set to True? I think it has to do with atexit not closing the process?

[dave7280]

Same here... and it even leaks on autoclose = False