Not using a fake user-agent as the default one. Making user-agent configurable.

kazet commented 8 months ago

Description

Thanks for an awesome tool! In my opinion a fake user/agent should not be the default one, therefore I suggest changing the default user-agent to humble. Besides that, I have made the user-agent configurable.

Type of change

Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Running a simple HTTP server and testing Humble both with ua argument set and unset, checking what is the User-Agent header.

Checklist:

[ ] My code follows the style guidelines of this project - used flake8, where are the guidelines?
[X] I have performed a self-review of my own code
[N/A] I have commented my code, particularly in hard-to-understand areas
[N/A] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings - used flake8
[ ] I have added tests that prove my fix is effective or that my feature works - where are the tests?
[ ] New and existing unit tests pass locally with my changes - where are the tests?
[N/A] Any dependent changes have been merged and published in downstream modules
[X] I have checked my code and corrected any misspellings

rfc-st commented 8 months ago

Hi!,

Thanks for your contribution; yes, the option to send a customized User-Agent is something I had been thinking about for some time: in the tests I have done these years many of the requested URLs expect (at least) that this header is sent with a 'recognizable' content to return a valid response (ex. combination of OS + Browser + Engine, etc) and to prevent the request from being associated with a web crawler, bot or scanner.

In fact, I think that sending only the User-Agent in an HTTP request, even if it is well-formed and 'real', is clearly insufficient for some of these URLs ... but that's another issue that I will also improve when I have time :).

By the way: the User-Agent that includes 'humble' by default ('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36') is real :), you can check it by parsing it in https://explore.whatismybrowser.com/useragents/parse/

I have to check calmly your PR, as I am convinced that if the User-Agent is simply the literal 'humble' there will be problems with some URLs (being in this case a minimal value unrelated to any OS/Browser).

Give me a few days, please, to think about this (I also want to include some minimal validation of the customized User-Agent) and we'll talk.

Thank you.

P.S: good luck with Artemis! :).

Best regards,

rfc-st commented 8 months ago

Hello!,

Please have a look at: https://github.com/rfc-st/humble/commit/b2a0e14e1294732779153ade4417faf9050474c1

I think it may be of interest to you :):

.- New parameter '-ua'. .- It accepts a number. .- Which is associated with the entry in the User-Agent file (_'additional/useragents.txt' file). .- I have included several validations: default values, '0' value, etc.

Best regards,

rfc-st / humble