RafaelWO / unparallel

Create async web requests via Python in no time.
https://rafaelwo.github.io/unparallel/
MIT License
10 stars 0 forks source link

Change: Mitigate timeouts by adding Semaphore #128

Closed RafaelWO closed 8 months ago

RafaelWO commented 8 months ago

Description

This PR adds a semaphore object to synchronize the web requests with a default value of 1000 (magic number). The intuition behind that is if you do a ton of web requests, all async tasks + TCP requests are created immediately and their "timer" is started. If we restrict the number of parallel web requests, they can wait until others are completed without timing out.

I chose 1000 for the semaphore because it is unlikely that HTTPX will open more than 1k TCP connections and therefore this should not slow down HTTPX.

Related Issue

n/a

Type of Change

Checklist