csaf_downloader fails on cert-bund.de?

[SystemSh0cker]

Was trying to fetch the CSAF-Files, but it's failing. Am I doing something wrong?

csaf_downloader.exe -v cert-bund.de
2023/09/08 14:23:29 [GET]: https://cert-bund.de/.well-known/csaf/provider-metadata.json
2023/09/08 14:23:29 [GET]: https://cert-bund.de/.well-known/security.txt
2023/09/08 14:23:30 [GET]: https://csaf.data.security.cert-bund.de
2023/09/08 14:23:30 Loading provider-metadata.json for "cert-bund.de": fetching "https://csaf.data.security.cert-bund.de" failed: Get "https://csaf.data.security.cert-bund.de": EOF
2023/09/08 14:23:30 error: no valid provider-metadata.json found for 'cert-bund.de'

[bernhardreiter]

Am I doing something wrong?

No, there is something wrong, I have the same issue. Thanks for reporting it!

[s-l-teichmann]

Okay found the issue:

Somewhere on the path of finding the "provider-metadata.json" (needed to find the advisories) the downloader tries to resolve the URL https://wid.cert-bund.de/.well-known/csaf/provider-metadata.json.

If you curl this everything is fine and you get your document. If you try to fetch it with the downloader it fails.

Turns out that the server is looking at the user agent and does not serve content to the standard Go http client.

Solution: You have to set the User-Agent manually:

csaf_downloader --header User-Agent:curl/7.55.1 bsi.de

@tschmidtb51 This is not nice

@SystemSh0cker Hope that helps

[SystemSh0cker]

Thanks! It's working with the User-Agent. There is already another Issue #376 but labeled with "wontfix".

Will be hard to fetch the Documents if User-Agents are sometimes required ... but CURL for example might be blocked (Akamai)

[bernhardreiter]

@SystemSh0cker that some User-Agents are not served, means that cert-bund.de currently is not CSAF 2.0 compatible.

[bernhardreiter]

I've notified our contacts at the BSI. Here is a request with httpie that shows the problem without our downloader

https -v https://wid.cert-bund.de/.well-known/csaf/provider-metadata.json User-Agent:Go-http-client/1.1
[..]
RemoteDisconnected('Remote end closed connection without response')

[bernhardreiter]

Can we improve the diagnostic messages somehow?

It seems that the original message has a point in

curl --user-agent "Go-http-client/1.1" https://csaf.data.security.cert-bund.de
curl: (52) Empty reply from server

which works in a browser... so there is a second URL combination which does not work.

And the other problem is not noted (that after the redirect from https://cert-bund.de/.well-known/csaf/provider-metadata.json there is also an EOF).

[s-l-teichmann]

Can we improve the diagnostic messages somehow?

At debug level we could add a callback which logs the redirections.

[cintek]

This commit is a suggestion how to implement the logging.

[tschmidtb51]

The original problem is now solved. We need to check on the improved logging stuff.

[tschmidtb51]

The issue was solved.