lycheeverse / lychee

⚡ Fast, async, stream-based link checker written in Rust. Finds broken URLs and mail addresses inside Markdown, HTML, reStructuredText, websites and more!
https://lychee.cli.rs
Apache License 2.0
2.23k stars 136 forks source link

Add support for custom headers in input processing #1561

Open mre opened 2 weeks ago

mre commented 2 weeks ago

Previously, we didn't pass (auth) headers to the initial set of inputs -- only to the links which were discovered in the initial input set. That's inconsistent and led to some confusion.

This PR fixes that. There is probably a nicer design for input parsing; the constructor has quite a few "dummy" params now, which tend to be set to the same values. I guess we can refactor that to, say, introduce the notion of an InputBuilder if it becomes to tedious to set these values. I guess we should decouple input parsing for the different input types in general because header handling is only needed for web requests and not for file inputs.

For now, I just introduced a helper method Input::from_value.

This fixes #1441.