search

opsdisk / pagodo

pagodo (Passive Google Dork) - Automate Google Hacking Database scraping and searching
GNU General Public License v3.0
2.67k stars 487 forks source link

error code 429 #33

Closed Yoomarksa closed 4 years ago

Yoomarksa commented 4 years ago

Why is it happening? HTTP Error code 429?

This happens often and no file is created.

opsdisk commented 4 years ago

Hi @Yoomarksa - It would help if you provided some more information. Can you provide the full command and switches you are using? As well as a screen shot?

Yoomarksa commented 4 years ago

umm... I'm excute command ./pagodo And this happening Error code 429

Error with dork: site:koica.go.kr intext:confidential filetype:doc | filetype:hwp | filetype:pdf | filetype:ppt - filename:code-of-conduct* [-] EXCEPTION: HTTP Error 429: Too Many Requests

opsdisk commented 4 years ago

Running ./pagodo by itself won't work, you need to provide at least the domain (-d) and dork file (-g). Can you provide me the entire command you are trying to run?

You may need to increase your -e and -j values since Google may be throttling you since you are making requests too fast.

etslab commented 4 years ago

Please see below

$ more /var/tmp/contents/sample_syntax.txt
inurl:what-is-collaboration.html
intext:"Learn how collaboration technologies"
$

Error:

$ /usr/bin/proxychains /usr/bin/python3 /var/tmp/pagodo/pagodo.py -d cisco.com -g /var/tmp/contents/sample_syntax.txt -e 50.0 -j 2.1`

`ProxyChains-3.1 (http://proxychains.sf.net)
[*] Initiation timestamp: 20191004_062524
[*] Search ( 1 / 2 ) for Google dork [ site:cisco.com inurl:what-is-collaboration.html ] and waiting 143.1790918133059 seconds between searches using User-Agent 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.697.0 Safari/534.24'
|DNS-request| www.google.com
|D-chain|-<>-127.0.0.1:9050-<>-127.0.0.1:9051-<--denied
|D-chain|-<>-127.0.0.1:9050-<><>-4.2.2.2:53-<><>-OK
|DNS-response| www.google.com is 216.58.206.164
|D-chain|-<>-127.0.0.1:9050-<>-127.0.0.1:9051-<--denied
|D-chain|-<>-127.0.0.1:9050-<><>-216.58.206.164:443-<><>-OK
|DNS-request| www.google.com
|D-chain|-<>-127.0.0.1:9050-<>-127.0.0.1:9051-<--denied
|D-chain|-<>-127.0.0.1:9050-<><>-4.2.2.2:53-<><>-OK
|DNS-response| www.google.com is 172.217.10.4
|D-chain|-<>-127.0.0.1:9050-<><>-172.217.10.4:443-<><>-OK
|DNS-request| www.google.com
|D-chain|-<>-127.0.0.1:9050-<>-127.0.0.1:9051-<--denied
|D-chain|-<>-127.0.0.1:9050-<><>-4.2.2.2:53-<><>-OK
|DNS-response| www.google.com is 172.217.10.4
|D-chain|-<>-127.0.0.1:9050-<><>-172.217.10.4:443-<><>-OK
[-] Error with dork: inurl:what-is-collaboration.html
[-] EXCEPTION: HTTP Error 429: Too Many Requests
[*] Search ( 2 / 2 ) for Google dork [ site:cisco.com intext:"Learn how collaboration technologies" ] and waiting 111.26572429799654 seconds between searches using User-Agent 'Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2a1pre) Gecko/20090428 Firefox/3.6a1pre'
|DNS-request| www.google.com
|D-chain|-<>-127.0.0.1:9050-<>-127.0.0.1:9051-<--denied
|D-chain|-<>-127.0.0.1:9050-<><>-4.2.2.2:53-<><>-OK
|DNS-response| www.google.com is 172.217.10.4
|D-chain|-<>-127.0.0.1:9050-<><>-172.217.10.4:443-<><>-OK
|DNS-request| www.google.com
|D-chain|-<>-127.0.0.1:9050-<>-127.0.0.1:9051-<--denied
|D-chain|-<>-127.0.0.1:9050-<><>-4.2.2.2:53-<><>-OK
|DNS-response| www.google.com is 172.217.10.4
|D-chain|-<>-127.0.0.1:9050-<><>-172.217.10.4:443-<><>-OK
|DNS-request| www.google.com
|D-chain|-<>-127.0.0.1:9050-<>-127.0.0.1:9051-<--denied
|D-chain|-<>-127.0.0.1:9050-<><>-4.2.2.2:53-<><>-OK
|DNS-response| www.google.com is 172.217.10.4
|D-chain|-<>-127.0.0.1:9050-<><>-172.217.10.4:443-<><>-OK
[-] Error with dork: intext:"Learn how collaboration technologies"
[-] EXCEPTION: HTTP Error 429: Too Many Requests
[*] Total dorks found: 0
[*] Completion timestamp: 20191004_062949
[+] Done!
$
opsdisk commented 4 years ago

Thanks for the datapoint @etslab . What does your proxychains.conf file look like? Are you SSHing into 2 different boxes and setting up a SOCKS proxy using D 9050 and -D 9051? The denied messages make me think something is wrong with your proxychains setup and your requests are leaving your host box's source IP and getting blocked by Google.

For both of you @etslab and @Yoomarksa , I've found you may have to wait a few minutes before trying again because Google thinks you're (rightfully) a bot.

etslab commented 4 years ago

Thanks @opsdisk for the prompt reply. No, I am not using any socks proxy, using tor for the proxychains

sudo apt-get install proxychains

sudo apt-get install tor

sudo service tor start

and proxychains config FYR

$ more /etc/proxychains.conf
round_robin
chain_len = 1
proxy_dns
remote_dns_subnet 224
tcp_read_time_out 15000
tcp_connect_time_out 8000
[ProxyList]
socks4 127.0.0.1 9050
socks4 127.0.0.1 9051`

Same configuration working fine my localhost but not from cloud instance. Wondering what could be a issue? Give me some idea

etslab commented 4 years ago

I have tried with free socks4 proxy available on the internet and following in the output

ProxyChains-3.1 (http://proxychains.sf.net)
[*] Initiation timestamp: 20191004_134518
[*] Search ( 1 / 2 ) for Google dork [ site:cisco.com inurl:what-is-collaboration.html ] and waiting 87.70113815378582 seconds between searches using User-Agent 'Mozilla/5.0 (X11; U; Linux i686; pl-PL; rv:1.9.0.2) Gecko/2008092313 Ubuntu/9.25 (jaunty) Firefox/3.8'
|DNS-request| www.google.com
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<>-103.250.16.27:44252-<><>-4.2.2.2:53-<><>-OK
|DNS-response| www.google.com is 172.217.26.228
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<>-103.250.16.27:44252-<><>-172.217.26.228:443-<><>-OK
|DNS-request| www.google.com
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<>-103.250.16.27:44252-<--timeout
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<><>-4.2.2.2:53-<><>-OK
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<><>-4.2.2.2:53-<><>-OK
|DNS-response| www.google.com is 172.217.169.132
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<>-103.250.16.27:44252-<--timeout
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<><>-172.217.169.132:443-<><>-OK
|DNS-request| www.google.com
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<>-103.250.16.27:44252-<--timeout
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<><>-4.2.2.2:53-<><>-OK
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<><>-4.2.2.2:53-<><>-OK
|DNS-response| www.google.com is 172.217.168.4
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<><>-172.217.168.4:443-<--timeout
[-] Error with dork: inurl:what-is-collaboration.html
[-] EXCEPTION: <urlopen error [Errno 111] Connection refused>
[*] Search ( 2 / 2 ) for Google dork [ site:cisco.com intext:"Learn how collaboration technologies" ] and waiting 80.37855464943297 seconds between searches using User-Agent 'Mozilla/5.0 (Windows; U; Windows NT 5.1; hu-HU) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16'
|DNS-request| www.google.com
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<>-103.250.16.27:44252-<--timeout
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<><>-4.2.2.2:53-<><>-OK
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<><>-4.2.2.2:53-<><>-OK
|DNS-response| www.google.com is 172.217.169.132
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<><>-172.217.169.132:443-<><>-OK
|DNS-request| www.google.com
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<>-103.250.16.27:44252-<><>-4.2.2.2:53-<><>-OK
|DNS-response| www.google.com is 172.217.167.164
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<><>-172.217.167.164:443-<><>-OK
|DNS-request| www.google.com
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<>-103.250.16.27:44252-<><>-4.2.2.2:53-<><>-OK
|DNS-response| www.google.com is 172.217.160.196
|D-chain|-<>-127.0.0.1:9050-<>-212.42.116.29:4145-<><>-172.217.160.196:443-<><>-OK
[-] Error with dork: intext:"Learn how collaboration technologies"
[-] EXCEPTION: HTTP Error 429: Too Many Requests
[*] Total dorks found: 0
[*] Completion timestamp: 20191004_134941
[+] Done!

I am not familiar with proxychains. Please let me know if need to setup myown socks proxy to make this work?

opsdisk commented 4 years ago

@etslab Google may track active Tor exit nodes / public proxies and block them (or require you answer a captcha which I've seen before). Sounds like your issue is more around using proxychains, and not necessarily with pagodo (although it could be, but proxychains complicates troubleshooting). If you want to learn more about SSH and SOCKS proxies, this isn't the best ticket to troubleshoot/teach that and I'd gently direct you to this book (https://cph.opsdisk.com/) which covers that and is free if you are a student.

opsdisk commented 4 years ago

Going to close this one out.