Silent failure if -random-agent is the first argument

[njbooher]

Describe the bug

HTTPX fails silently if -random-agent is the first argument

Environment details

# httpx -version

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/              v1.1.3

                projectdiscovery.io

Use with caution. You are responsible for your actions.
Developers assume no liability and are not responsible for any misuse or damage.
[INF] Current Version: v1.1.3
# go version
go version go1.14.7 linux/amd64
# uname -a
Linux ffuf 4.15.0-96-generic #97-Ubuntu SMP Wed Apr 1 03:25:46 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I'm using the download from here; https://github.com/projectdiscovery/httpx/releases/download/v1.1.3/httpx_1.1.3_linux_amd64.zip

Error details

This command works

echo 'google.com' | httpx -H 'User-Agent: asdf' -include-response -json -verbose -debug -random-agent false

Output:

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/              v1.1.3

                projectdiscovery.io

Use with caution. You are responsible for your actions.
Developers assume no liability and are not responsible for any misuse or damage.
{"timestamp":"2021-10-17T16:08:14.145923198Z","request":"GET / HTTP/1.1\r\nHost: google.com\r\nUser-Agent: asdf\r\nAccept-Charset: utf-8\r\nAccept-Encoding: gzip\r\n\r\n","response-header":"HTTP/1.1 301 Moved Permanently\r\nConnection: close\r\nContent-Length: 220\r\nAlt-Svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"\r\nCache-Control: public, max-age=2592000\r\nContent-Type: text/html; charset=UTF-8\r\nDate: Sun, 17 Oct 2021 16:08:14 GMT\r\nExpires: Tue, 16 Nov 2021 16:08:14 GMT\r\nLocation: https://www.google.com/\r\nServer: gws\r\nX-Frame-Options: SAMEORIGIN\r\nX-Xss-Protection: 0\r\n\r\n","scheme":"https","port":"443","path":"/","body-sha256":"5b61b0c2032b4aa9519d65cc98c6416c12415e02c7fbbaa1be5121dc75162edb","header-sha256":"aa12777a9c121be9484e007ee535a039d60d2675b1c072b6cbb322ff9b2cf3ac","a":["142.250.80.46"],"url":"https://google.com:443","input":"google.com","location":"https://www.google.com/","title":"301 Moved","webserver":"gws","response-body":"\u003cHTML\u003e\u003cHEAD\u003e\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003cTITLE\u003e301 Moved\u003c/TITLE\u003e\u003c/HEAD\u003e\u003cBODY\u003e\n\u003cH1\u003e301 Moved\u003c/H1\u003e\nThe document has moved\n\u003cA HREF=\"https://www.google.com/\"\u003ehere\u003c/A\u003e.\r\n\u003c/BODY\u003e\u003c/HTML\u003e\r\n","content-type":"text/html","method":"GET","host":"142.250.80.46","content-length":220,"status-code":301,"response-time":"36.699954ms","failed":false} 

This fails silently with no hint as to why

echo 'google.com' | httpx -random-agent false -H 'User-Agent: asdf' -include-response -json -verbose -debug

Output:

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/              v1.1.3

                projectdiscovery.io

Use with caution. You are responsible for your actions.
Developers assume no liability and are not responsible for any misuse or damage.
https://google.com

[ehsandeep]

@njbooher I guess the right command to use would be echo 'google.com' | httpx -H 'User-Agent: asdf' -include-response -json, if you are setting custom User-Agent using -H flag, the random UA will be disabled internally, so using -random-agent false is not expected.

but I can see -random-agent might cause confusion and as it's enabled as default, we will be removing this CLI flag in future updates.

[njbooher]

Ok, I'll go ahead and remove -random-agent from my scripts. Thanks!