RafaelWO / unparallel

Create async web requests via Python in no time.
https://rafaelwo.github.io/unparallel/
MIT License
10 stars 0 forks source link

Fix async synchronization and timeouts #197

Closed RafaelWO closed 4 months ago

RafaelWO commented 4 months ago

Description

After some benchmarking, it seems that setting the semaphore value to the same value as max_connections gives better performance and fewer timeout errors. If wanted, users can change the semaphore value via a keyword argument.

The plot below shows 5 different runs of Unparallel doing 2000 HTTP requests with varying max_connections (x-axis) and semaphore value which is defined by max_connections * semaphore_fact, i.e. for max_connections=300 and semaphore_fact=2 the semaphore value is 600. The y-axis shows the run time in seconds.

benchmarking results for different semaphore values

The data shows that if the semaphore value is greater than the maximum connections, the run time of finishing the 2000 requests increases drastically.


Additionally, some dependencies are upgraded due to CVEs. A policy config for safety is added to ignore a vulnerability in jinja2 that can't be fixed.

Changes

Related Issue

Relates to #157

Type of Change

Checklist