FlorianREGAZ / Python-Tls-Client

Advanced HTTP Library
MIT License
678 stars 135 forks source link

memory leak #59

Open xiamoplus opened 1 year ago

xiamoplus commented 1 year ago

Hello, I saw that the memory leak was fixed by a recent pull request, but I still get an error from it. Any idea on how to avoid it? thx

OxynDev commented 1 year ago

Update lib pip install --force-reinstall tls-client

DaBossCoda commented 1 year ago

Also, having a memory leak issue. Can't seem to fix it :/

sumeshmurali commented 1 year ago

The memory leak is still present. The issue can be reproduced with the following code. I have hosted a local httpbin server and sent some large random data as JSON to the library. The memory in the first iteration is around 36 MB which gradually increases to more than 250MB.

import tls_client

from memory_profiler import profile

@profile
def send_requests():
    session = tls_client.Session(
        client_identifier="chrome112",
        random_tls_extension_order=True
    )
    url = 'http://0.0.0.0:8080/anything'
    random_data = ["a"] * 10 ** 4
    res = session.post(url=url, json=random_data)
    json = res.json()
    print(len(json))

for i in range(100000):
    send_requests()
kastger commented 4 months ago

@sumeshmurali Haven't checked how tls_client handles session closing, but locally I was able to reproduce the issue with your code. Memory for me increased from 55MB -> 283MB (10000) requests. However, after adding session close the memory seems to stabilize. Could you try it on your side and confirm or deny if this works?

Edit: After testing it on a real and relatively heavy page (15MB, used GET instead of POST) the leak seems to still be present even with session.close().

import tls_client

from memory_profiler import profile

@profile
def send_requests():
    session = tls_client.Session(
        client_identifier="chrome112",
        random_tls_extension_order=True
    )
    url = 'http://localhost:8000/anything'
    random_data = ["a"] * 10 ** 4
    try:
        res = session.post(url=url, json=random_data)
        print(res.status_code)
    finally:
        session.close()

for i in range(10000):
    send_requests()
OxynDev commented 3 months ago

Try add force_http1 tls_client.Session(client_identifier="firefox119",force_http1=True) In golang version it help me solv this issue

YABOIpy commented 3 months ago

@OxynDev the issue being talked about is a memory leak. forcing the client to use http/1.1 would work. just not smart if you're trying to replicate a browser which this client is made for. depending on what backend you're sending these requests to. most will support this proto, but likely stuff wont work as intended. as servers will expect a browser to do http requests over newer protos