search

projectdiscovery / httpx

httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.
https://docs.projectdiscovery.io/tools/httpx
MIT License
7.61k stars 824 forks source link

Encountering a Strange Bug While Running Golang Compiled Program from Python using subprocess.run() #1085

Closed Miracles666 closed 8 months ago

Miracles666 commented 1 year ago

I encountered a very strange bug. I want to use Python to call a Golang compiled httpx program by running test.py script. Here's the code:

java Copy code import subprocess

cmd = "sudo /home/miracles/xxxx/client/subdomain/thirdparty/httpx/httpx -t 30 -rl 60 -fr -ec -sc -cl -title -td -l /home/miracles/jx-scan/client/subdomain/results/temp/domains_tmp_1681376467_XpFhcHfcOJ.txt "

subprocess.run(cmd, shell=True) When I run the program by right-clicking on it in PyCharm and selecting 'run', it gets stuck and doesn't work, like this:

   __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

  projectdiscovery.io

[INF] Current httpx version v1.2.9 (latest)

It doesn't return any results. However, when I run the code using the command python3 test.py, it successfully returns the expected results:

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

                projectdiscovery.io

[INF] Current httpx version v1.2.9 (latest)
http://www.xxxx.com.cn [302,301,302,200] [126084] [xxxxxx务] [https://www.xxx.com/official/home-a?channel_id=1]

I noticed that this issue only occurs with binaries from Project Discovery, as their other products like naabu also have the same problem."

ananan commented 1 year ago

I got the same issue, had you fixed it?

ehsandeep commented 1 year ago

@ananan @Miracles666 using -no-stdin option should resolve this issue.

ananan commented 1 year ago

need add shell=False to subprocess.run too

Mzack9999 commented 8 months ago

Unfortunately it's related to the particular way the os buffers stdin. All the tools attempt to collect all the targets from various input sources and deduplicate them. Shell=True implies that the command is executed as a child process of the default shell, which indirectly connects stdin/stdout/stderr to it via Popen.communicate, hence httpx detects the presence of potential data via stdin. Since you are using an absolute path, as @ananan mentioned, you could set Shell=False, additionally I think you can remove sudo since httpx really doesn't need elevated privileges.

I'm closing the issue due to inactivity - Feel free to reopen in case you are still facing troubles