search

gocsaf / csaf

Tools to download or provide CSAF (Common Security Advisory Framework) documents.
https://csaf.io
42 stars 25 forks source link

Add concurrent downloads to downloader. #363

Closed s-l-teichmann closed 1 year ago

s-l-teichmann commented 1 year ago

Add --worker flag to downloader.

Some sites seems to limit the number of requests per time interval per http client. This PR adds a flag to download more than one advisory at once via multiple http clients to speed up the process. By default only two clients are used.

Furthermore this PR also adds the feature to stop the downloader by Ctrl-Cing.

When the ignoresigcheck flag is set the results of the signature checks are ignored.

Bumped Go to 1.20 to use errors.Join.

s-l-teichmann commented 1 year ago

Shouldn't we use a default with at least 2 to demonstrate this advantage (and get more testing of it)?

We could ... I'm a bit undecided how conservative our default setting should be. @tschmidtb51 ?

tschmidtb51 commented 1 year ago

Bumped Go to 1.20 to use errors.Join.

Please add an extra note for that to the release notes...

tschmidtb51 commented 1 year ago

We could ... I'm a bit undecided how conservative our default setting should be. @tschmidtb51 ?

In general, I liked the idea of doing it based on the available CPUs but I understand that this might be harder to explain... IMHO, we should set the default (at least) to 2 but allow to reduce it to 1.

s-l-teichmann commented 1 year ago

Bumped Go to 1.20 to use errors.Join.

Please add an extra note for that to the release notes...

Will do.

s-l-teichmann commented 1 year ago

We could ... I'm a bit undecided how conservative our default setting should be. @tschmidtb51 ?

In general, I liked the idea of doing it based on the available CPUs but I understand that this might be harder to explain... IMHO, we should set the default (at least) to 2 but allow to reduce it to 1.

Commit c98fafe0a1598c864519d9ff415d594aaa9b787c sets the default to two.