Exceptions in threads during scanning in Level 1 & 2

s0md3v / Photon

Incredibly fast crawler designed for OSINT.

GNU General Public License v3.0

10.96k stars 1.49k forks source link

Exceptions in threads during scanning in Level 1 & 2 #120

Closed 4k4xs4pH1r3 closed 5 years ago

4k4xs4pH1r3 commented 5 years ago

Exception in thread Thread-4694: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.__kwargs) File "photon.py", line 211, in extractor if is_link(link, processed, files): File "/usr/share/Photon/core/utils.py", line 41, in is_link is_file = url.endswith(BAD_TYPES) TypeError: endswith first arg must be str, unicode, or tuple, not list

s0md3v commented 5 years ago

I am unable to reproduce the issue, would it be possible to share the website you were crawling? Thanks.

TerrorFactor commented 5 years ago

Had the same issue with multiple sites, even with just -u="google.com" It does seem to happen only on python 2, python 3 is ok.

s0md3v commented 5 years ago

Looks like you are using a large number of threads or your network connection is not very fast. Please use a higher value of timeout as follows

python photon.py -u "http://example.com --timeout=20

The default timeout value is 6.

TerrorFactor commented 5 years ago

Seems unlikely. From speedtest.net: ping = 0 ms, download = 885 mbps, upload = 939 mbps. Threads is default (not specified), adding "--timeout=20" doesn't help. Tried also with --timeout=2000, also no effect.

Using Python 2.7.15+ on Ubuntu in Windows Subsytem for Linux.

s0md3v commented 5 years ago

That's strange. I had a half mind about dropping support for python 2. Looks like it needs to be done.

TerrorFactor commented 5 years ago

That's one way of course. It would definitely be more clear.