More output on errors with external links on debug log level - Githubissues

wjdp / htmltest

:white_check_mark: Test generated HTML for problems

MIT License

323 stars 54 forks source link

More output on errors with external links on debug log level #221

Open r4fterman opened 1 year ago

r4fterman commented 1 year ago

I am currently running into 403 and ExternalTimeout errors for some links on the internet I am having trouble to identify the root cause. Using curl for the same links is working fine.

There is the option to set log-level to debug but this is not much of a help in indentifying the root of the failing check. Output is only.

E.g.: Status 403

[0m[35m  fresh --- mysql-jdbc.html --> https://dev.mysql.com/downloads/connector/j/
[0m[34m  hitting --- mysql-jdbc.html --> https://dev.mysql.com/downloads/connector/j/
[0m[31m  Non-OK status: 403 --- mysql-jdbc.html --> https://dev.mysql.com/downloads/connector/j/

Timeout

[0m[35m  fresh --- custom.html --> https://www.dnb.com/de-de/produkte-services/dun-bradstreet/dnb-duns-nummer
[0m[34m  hitting --- custom.html --> https://www.dnb.com/de-de/produkte-services/dun-bradstreet/dnb-duns-nummer
[0m[31m  request exceeded our ExternalTimeout --- custom.html --> https://www.dnb.com/de-de/produkte-services/dun-bradstreet/dnb-duns-nummer

On DEBUG level I expect the following Information to be added:

Final URL that is crawled (there might some adaptions to the URL as seen. in the code)
Request method is used (from code it seems to be GET)
Response code
Response content