Golang | Embedding a WaitGroup within a custom HTTP client

atc0005 / learn

Various topics that I want to learn more about, either for professional development or for my own use

0 stars 0 forks source link

Golang | Embedding a WaitGroup within a custom HTTP client #100

Open atc0005 opened 3 years ago

atc0005 commented 3 years ago

There are a number of items to study further, but these stand out:

embedding a WaitGroup into a type which has multiple methods used for spinning off recursive goroutines
calling Wait() within a wrapper method for the recursive goroutine generating method

refs https://www.youtube.com/watch?v=zMgpEcFN1og

atc0005 commented 3 years ago

Worth noting: This example does not (seem to) apply an upper limit on spawned goroutines. While the client is likely perfectly happy with this arrangement, the remote server being crawled may not be fine with being hammered by by the crawler, especially if the server is not tuned for the load.

See https://github.com/atc0005/check-cert/issues/103#issuecomment-725395798 for an approach using channels with a set queue depth to limit the maximum number of concurrent connections.