JuliaLang / Downloads.jl

MIT License
89 stars 34 forks source link

add concurrent connection semaphore to prevent DNS resolve races #205

Closed StefanKarpinski closed 1 year ago

StefanKarpinski commented 1 year ago

The libcurl spawns a new thread for each DNS resolution that it needs to do, which can cause problems when too many connections are started at the same time. This adds a semaphore which is acqured when a connection starts and gets released once the actual request starts. This limits the number of concurrent requests that are in the connecting state to the size of the semaphore (16). Hopefully this prevents the "getaddrinfo() thread failed" errors that occur sometimes.

codecov[bot] commented 1 year ago

Codecov Report

Merging #205 (ab06f67) into master (0733701) will increase coverage by 0.46%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #205      +/-   ##
==========================================
+ Coverage   92.22%   92.68%   +0.46%     
==========================================
  Files           5        5              
  Lines         527      547      +20     
==========================================
+ Hits          486      507      +21     
+ Misses         41       40       -1     
Impacted Files Coverage Δ
src/Curl/Curl.jl 100.00% <ø> (ø)
src/Curl/Easy.jl 96.04% <100.00%> (+0.27%) :arrow_up:
src/Curl/Multi.jl 94.30% <100.00%> (+0.09%) :arrow_up:
src/Curl/utils.jl 75.00% <0.00%> (+4.16%) :arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

DilumAluthge commented 1 year ago

Is this backportable?

StefanKarpinski commented 1 year ago

Maybe, I'm not sure how clean it would be. I'll have to go through and see what changes we can backport at some point.