add configuration option for additional request headers

[dardanbujupaj]

Pull Request

Related issue

none

What does this PR do?

We're using scrapix to index our documentation. Some parts of the documentation are protected, so we need a way to send some authentication information with the crawler requests. This PR adds a new configuration property additional_request_headers which are applied to each request in a pre navigation hook. This way we're able to either send an Authorization header (basic, bearer etc.) or a custom auth header (e.g. x-access-key), so it should be quite flexible for any usecase.

• add configuration option for additional_request_headers
• add a pre navigation hook to add those headers to the request

[dardanbujupaj]

PS: we're running this from our fork, but I thought the changes might as well be relevant to this repo. feel free to merge or close the PR as you see fit 🙃

[brunoocasali]

Hey @dardanbujupaj, thanks a lot for your contribution!!

Can you include some examples of how you can use this feature in the README?

[dardanbujupaj]

Can you include some examples of how you can use this feature in the README?

@brunoocasali Sure, I'll add something this evening.

[dardanbujupaj]

@brunoocasali added the configuration property description to the readme with a short example.

Let me know if there's anything else to improve!

[meili-bors[bot]]

Configuration problem: bors.toml: not found