search

lycheeverse / lychee

⚡ Fast, async, stream-based link checker written in Rust. Finds broken URLs and mail addresses inside Markdown, HTML, reStructuredText, websites and more!
https://lychee.cli.rs
Apache License 2.0
2.21k stars 134 forks source link

Security: restrict custom HTTP request headers to specific URL patterns #1298

Open sanmai-NL opened 1 year ago

sanmai-NL commented 1 year ago

Currently, the --header CLI and configuration parameter does not distinguish between target URLs. If the user were to supply a crucial secret in there, in order to make some URLs accessible during CI, then this secret would leak to all other hosts for which URLs are found.

Solution design

Rather than implementing URL/header mapping logic in Lychee, I propose to separate this concern into an, if you so choose, external tool like a proxy. If Lychee were to have proxying support, complex logic, mappings, analysis, flows, etc. can be configured through a proxy. If common use cases are documented in a how-to style within the Lychee docs, the value for the user would not be much less than with a native implementation by Lychee (one could argue, much more indeed).

sanmai-NL commented 1 year ago

An example of a tool that could support this functionality: https://docs.mitmproxy.org/stable/overview-features/#modify-headers

mre commented 9 months ago

I like the modify_headers syntax you linked to. We could add something like this. Just to clarify, this doesn't require a proxy, but rather a way to pass these headers to reqwest, our HTTP request client, right?