projectdiscovery / httpx

httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.
https://docs.projectdiscovery.io/tools/httpx
MIT License
7.53k stars 822 forks source link

Buildup of uncollectible garbage when using `Client` and `AsyncClient` #1885

Closed tsmith023 closed 1 month ago

tsmith023 commented 1 month ago

httpx version:

0.27.0

Current Behavior:

When using the Client and AsyncClient objects in the context manager pattern, there seems to be a buildup of uncollectible garbage from their usage. Strangely, the number of uncollectible objects identified by the GC changes depending on whether the return of the client request is assigned to a variable or not.

Expected Behavior:

All objects created by the httpx library should be garbage collectable so as to avoid memory leakage.

Steps To Reproduce:

Sync using response:

def sync() -> None:
    gc.set_debug(gc.DEBUG_SAVEALL)
    with httpx.Client() as client:
        res = client.get("https://www.google.com")
        print("Status: ", res.status_code)
    gc.collect()
    print("Uncollectible Garbage: ", len(gc.garbage))

leads to:

Status:  200
Uncollectible Garbage:  5

Sync without using response:

def sync() -> None:
    gc.set_debug(gc.DEBUG_SAVEALL)
    with httpx.Client() as client:
        client.get("https://www.google.com")
    gc.collect()
    print("Uncollectible Garbage: ", len(gc.garbage))

leads to:

Uncollectible Garbage:  74

Async using response:

async def async_() -> None:
    gc.set_debug(gc.DEBUG_SAVEALL)
    async with httpx.AsyncClient() as client:
        res = await client.get("https://www.google.com")
        print("Status: ", res.status_code)
    gc.collect()
    print("Uncollectible Garbage: ", len(gc.garbage))

leads to:

Status:  200
Uncollectible Garbage:  37

Async without using response:

async def async_() -> None:
    gc.set_debug(gc.DEBUG_SAVEALL)
    async with httpx.AsyncClient() as client:
        await client.get("https://www.google.com")
    gc.collect()
    print("Uncollectible Garbage: ", len(gc.garbage))

leads to:

Uncollectible Garbage:  83

Anything else:

I may be misunderstanding the action of Python's Garbage Collector as this is the first time that I'm digging into it to understand the problem I'm facing. So if there's some aspect that I may be missing then it would be great to understand that in the context of this issue! Cheers 😁

tsmith023 commented 1 month ago

Wrong repo lol