search

wkeeling / selenium-wire

Extends Selenium's Python bindings to give you the ability to inspect requests made by the browser.
MIT License
1.9k stars 254 forks source link

When I open a website with websocket (wss) connection using selenium-wire, the memory footprint keeps growing #552

Open hokkz opened 2 years ago

hokkz commented 2 years ago 
================================: 2022-05-26 15:52:38.255177
/Users/zonst/2217/py3/lib/python3.8/site-packages/OpenSSL/SSL.py:1769: size=18.8 MiB, count=4783, average=4117 B
/Users/zonst/2217/py3/lib/python3.8/site-packages/seleniumwire/thirdparty/mitmproxy/net/tcp.py:145: size=10.7 MiB, count=2707, average=4129 B
================================: 2022-05-26 15:52:41.642926
/Users/zonst/2217/py3/lib/python3.8/site-packages/OpenSSL/SSL.py:1769: size=29.9 MiB, count=7623, average=4119 B
/Users/zonst/2217/py3/lib/python3.8/site-packages/seleniumwire/thirdparty/mitmproxy/net/tcp.py:145: size=17.1 MiB, count=4341, average=4126 B
================================: 2022-05-26 15:52:45.026702
/Users/zonst/2217/py3/lib/python3.8/site-packages/OpenSSL/SSL.py:1769: size=37.4 MiB, count=9520, average=4121 B
/Users/zonst/2217/py3/lib/python3.8/site-packages/seleniumwire/thirdparty/mitmproxy/net/tcp.py:145: size=21.5 MiB, count=5472, average=4128 B
================================: 2022-05-26 15:52:48.290644
/Users/zonst/2217/py3/lib/python3.8/site-packages/OpenSSL/SSL.py:1769: size=46.5 MiB, count=11816, average=4122 B
/Users/zonst/2217/py3/lib/python3.8/site-packages/seleniumwire/thirdparty/mitmproxy/net/tcp.py:145: size=27.1 MiB, count=6894, average=4129 B

The above is the information printing of my console memory usage. This situation will not appear in selenium. The following is to open the same environment and web page in selenium

================================: 2022-05-26 15:55:48.809998
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/tracemalloc.py:479: size=1664 B, count=23, average=72 B
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/tracemalloc.py:532: size=1352 B, count=22, average=61 B
================================: 2022-05-26 15:55:51.816318
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/tracemalloc.py:479: size=1720 B, count=24, average=72 B
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/tracemalloc.py:472: size=1344 B, count=24, average=56 B
================================: 2022-05-26 15:55:54.822369
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/tracemalloc.py:532: size=1792 B, count=31, average=58 B
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/tracemalloc.py:479: size=1720 B, count=24, average=72 B
================================: 2022-05-26 15:55:57.829736
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/tracemalloc.py:479: size=1720 B, count=24, average=72 B
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/tracemalloc.py:472: size=1392 B, count=25, average=56 B
wkeeling commented 2 years ago

Thanks for raising this. Yes websocket messages are currently stored in memory, so memory usage will grow over time as websocket messages are received. Calling del driver.requests will clear them out.

Storing websocket messages in memory was done to ease implementation initially, but we can probably improve on this now. I'll look at making a fix for the next release.

hokkz commented 2 years ago

Thank you for your attention. excited to put this issue on the version plan. I tried to use del driver.requests to clear them, but the result shows that the memory occupation brought by SSL.py tcp.py is not released, the following is the console information printing and code

for x in range(0, 20):
    print("================================:", datetime.datetime.now())
    print(len(driver.requests))
    snap = tracemalloc.take_snapshot()
    top_stats = snap.statistics('lineno')
    for stat in top_stats[:2]:
        print(stat)
    del driver.requests
    time.sleep(3)

================================: 2022-05-26 16:56:05.165671 0 /Users/zonst/2217/py3/lib/python3.8/site-packages/OpenSSL/SSL.py:1769: size=19.2 MiB, count=4888, average=4118 B /Users/zonst/2217/py3/lib/python3.8/site-packages/seleniumwire/thirdparty/mitmproxy/net/tcp.py:145: size=14.1 MiB, count=3587, average=4129 B ================================: 2022-05-26 16:56:08.416115 0 /Users/zonst/2217/py3/lib/python3.8/site-packages/OpenSSL/SSL.py:1769: size=23.0 MiB, count=5863, average=4118 B /Users/zonst/2217/py3/lib/python3.8/site-packages/seleniumwire/thirdparty/mitmproxy/net/tcp.py:145: size=16.9 MiB, count=4305, average=4128 B ================================: 2022-05-26 16:56:11.680175 0 /Users/zonst/2217/py3/lib/python3.8/site-packages/OpenSSL/SSL.py:1769: size=27.1 MiB, count=6891, average=4121 B /Users/zonst/2217/py3/lib/python3.8/site-packages/seleniumwire/thirdparty/mitmproxy/net/tcp.py:145: size=19.8 MiB, count=5041, average=4129 B ================================: 2022-05-26 16:56:14.896431 0 /Users/zonst/2217/py3/lib/python3.8/site-packages/OpenSSL/SSL.py:1769: size=30.8 MiB, count=7840, average=4122 B

wkeeling commented 2 years ago

Thanks for the additional info.

Aside from the memory usage, is the websocket connection functioning correctly - does the site work as expected? Just trying to establish whether the websocket connection is behaving or in a broken state.

hokkz commented 2 years ago

Yes, except that the memory usage continues to grow, the connection of web pages and web sockets is normal

hokkz commented 2 years ago

I guess it may not be the problem of selenium-wire, it is the reason why there are live videos in the website I visit, it may be the memory problem of the chrome browser

wkeeling commented 2 years ago

The problem may lie in mitmproxy which is the engine that Selenium Wire uses internally. I can try to test that by excluding Selenium Wire and using just mitmproxy. Are you able to share the URL?

hokkz commented 2 years ago

https://www.tiktok.com/live Try to enter any live room, it is recommended to open multiple live pages at the same time, it is easier to expose the problem

hokkz commented 2 years ago

I am not using selenium. When only using the chrome browser to open the page, the problem of continuous increase of memory still exists, so I don't think it is directly related to the related components of webdriver.

hokkz commented 2 years ago

I am not using selenium. When only using the chrome browser to open the page, the problem of continuous increase of memory still exists, so I don't think it is directly related to the related components of webdriver.

Sorry for my wrong statement, after I observed it again for a while, I found that the memory usage of chrome will stop increasing after a certain period, but using selenium or selenium-wire memory will increase without an upper limit

ArtemBernatskyy commented 9 months ago

were you able to fix this?