search

byt3bl33d3r / WitnessMe

Web Inventory tool, takes screenshots of webpages using Pyppeteer (headless Chrome/Chromium) and provides some extra bells & whistles to make life easier.
GNU General Public License v3.0
727 stars 109 forks source link

Error taking screenshot: expected string or bytes-like object #6

Closed ghost closed 4 years ago

ghost commented 4 years ago

I'll started WitnessMe with a .nessus file, but I receive the error

Error taking screenshot: expected string or bytes-like object

Screenshot: https://monosnap.com/file/OofiFk9CJ4JSLOr8dJX17x8rFon9kY

ghost commented 4 years ago

Note: same issue with text file with a list of IP's.

ghost commented 4 years ago

Other example:

python witnessme.py google.com

[I:pyppeteer.launcher] Browser listening on: ws://127.0.0.1:49937/devtools/browser/3cbd14a4-52b2-4e37-a0ea-e5791a7b5ae5 2019-11-12 16:54:38,022 [INFO] - witnessme.py: start_scan - Using 25 worker thread(s) 2019-11-12 16:54:38,991 [ERROR] - witnessme.py: worker - Error taking screenshot: net::ERR_EMPTY_RESPONSE at http://google.com:443 2019-11-12 16:54:39,004 [ERROR] - witnessme.py: worker - Error taking screenshot: net::ERR_SSL_PROTOCOL_ERROR at https://google.com:80 2019-11-12 16:54:40,171 [ERROR] - witnessme.py: worker - Error taking screenshot: expected string or bytes-like object 2019-11-12 16:54:40,181 [ERROR] - witnessme.py: worker - Error taking screenshot: expected string or bytes-like object 2019-11-12 16:54:42,682 [INFO] - witnessme.py: task_watch - total: 8, done: 4, pending: 4 2019-11-12 16:54:47,683 [INFO] - witnessme.py: task_watch - total: 8, done: 4, pending: 4 2019-11-12 16:54:52,683 [INFO] - witnessme.py: task_watch - total: 8, done: 4, pending: 4 2019-11-12 16:54:57,688 [INFO] - witnessme.py: task_watch - total: 8, done: 4, pending: 4 2019-11-12 16:55:02,693 [INFO] - witnessme.py: task_watch - total: 8, done: 4, pending: 4 2019-11-12 16:55:07,695 [INFO] - witnessme.py: task_watch - total: 8, done: 4, pending: 4 2019-11-12 16:55:12,700 [INFO] - witnessme.py: task_watch - total: 8, done: 4, pending: 4 2019-11-12 16:55:13,391 [INFO] - witnessme.py: worker - Task for url http://google.com:8080 timed out 2019-11-12 16:55:13,391 [INFO] - witnessme.py: worker - Task for url https://google.com:8080 timed out 2019-11-12 16:55:13,394 [INFO] - witnessme.py: worker - Task for url https://google.com:8443 timed out 2019-11-12 16:55:13,394 [INFO] - witnessme.py: worker - Task for url http://google.com:8443 timed out [I:pyppeteer.launcher] terminate chrome process...

byt3bl33d3r commented 4 years ago

You're using python 3 right? What OS are you running on ?

ghost commented 4 years ago

You're using python 3 right? What OS are you running on ?

Hi, yes python3. I’ve tried both MacOS and Ubuntu (latest) in a Docker container. Both gave same issue.

byt3bl33d3r commented 4 years ago

The only thing I can think of is try installing it outside of a docker container and see if it works there?

ghost commented 4 years ago

The only thing I can think of is try installing it outside of a docker container and see if it works there?

MacOs was not in a docker container, only Ubuntu was.

byt3bl33d3r commented 4 years ago

No clue. Works for me: image

Can you share the text file you're using?

ghost commented 4 years ago

Unfortunately I'm not allowed to share our public ip adresses, but it's just a bunch of public ip adresses.

It didn't work for google.com as in your example as well:

witnessme2

Anyway, thank you very much for your help and the efforts in building this tool.

byt3bl33d3r commented 4 years ago

@faun88 something that might help me figure this out.

Can you put the following right above line 120:

import traceback; traceback.print_exc()

Run it again and give me the traceback?

Thanks

ghost commented 4 years ago

@faun88 something that might help me figure this out.

Can you put the following right above line 120:

import traceback; traceback.print_exc()

Run it again and give me the traceback?

Thanks

Sure:

python witnessme.py www.google.com
[I:pyppeteer.launcher] Browser listening on: ws://127.0.0.1:50659/devtools/browser/40c461a1-4dd3-43e7-b09c-2bf0c008abf5
2019-11-12 22:52:31,551 [INFO] - witnessme.py: start_scan - Using 25 worker thread(s)
Traceback (most recent call last):
  File "witnessme.py", line 112, in worker
    r = await asyncio.wait_for(screenshot(url, page), timeout=args.timeout)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py", line 442, in wait_for
    return fut.result()
  File "witnessme.py", line 59, in screenshot
    "waitUntil": "networkidle0"
  File "/Users/dennis/.local/share/virtualenvs/WitnessMe-lSv3YZHu/src/pyppeteer-dev/pyppeteer/page.py", line 881, in goto
    raise PageError(result)
pyppeteer.errors.PageError: net::ERR_SSL_PROTOCOL_ERROR at https://www.google.com:80
2019-11-12 22:52:32,598 [ERROR] - witnessme.py: worker - Error taking screenshot: net::ERR_SSL_PROTOCOL_ERROR at https://www.google.com:80
Traceback (most recent call last):
  File "witnessme.py", line 112, in worker
    r = await asyncio.wait_for(screenshot(url, page), timeout=args.timeout)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py", line 442, in wait_for
    return fut.result()
  File "witnessme.py", line 59, in screenshot
    "waitUntil": "networkidle0"
  File "/Users/dennis/.local/share/virtualenvs/WitnessMe-lSv3YZHu/src/pyppeteer-dev/pyppeteer/page.py", line 881, in goto
    raise PageError(result)
pyppeteer.errors.PageError: net::ERR_CONNECTION_RESET at http://www.google.com:443
2019-11-12 22:52:32,600 [ERROR] - witnessme.py: worker - Error taking screenshot: net::ERR_CONNECTION_RESET at http://www.google.com:443
Traceback (most recent call last):
  File "witnessme.py", line 112, in worker
    r = await asyncio.wait_for(screenshot(url, page), timeout=args.timeout)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py", line 442, in wait_for
    return fut.result()
  File "witnessme.py", line 77, in screenshot
    'fullPage': True
  File "/Users/dennis/.local/share/virtualenvs/WitnessMe-lSv3YZHu/src/pyppeteer-dev/pyppeteer/page.py", line 1245, in screenshot
    mimeType, _ = mimetypes.guess_type(options['path'])
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/mimetypes.py", line 291, in guess_type
    return _db.guess_type(url, strict)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/mimetypes.py", line 116, in guess_type
    scheme, url = urllib.parse.splittype(url)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 983, in splittype
    match = _typeprog.match(url)
TypeError: expected string or bytes-like object
2019-11-12 22:52:34,420 [ERROR] - witnessme.py: worker - Error taking screenshot: expected string or bytes-like object
Traceback (most recent call last):
  File "witnessme.py", line 112, in worker
    r = await asyncio.wait_for(screenshot(url, page), timeout=args.timeout)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py", line 442, in wait_for
    return fut.result()
  File "witnessme.py", line 77, in screenshot
    'fullPage': True
  File "/Users/dennis/.local/share/virtualenvs/WitnessMe-lSv3YZHu/src/pyppeteer-dev/pyppeteer/page.py", line 1245, in screenshot
    mimeType, _ = mimetypes.guess_type(options['path'])
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/mimetypes.py", line 291, in guess_type
    return _db.guess_type(url, strict)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/mimetypes.py", line 116, in guess_type
    scheme, url = urllib.parse.splittype(url)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 983, in splittype
    match = _typeprog.match(url)
TypeError: expected string or bytes-like object
2019-11-12 22:52:35,376 [ERROR] - witnessme.py: worker - Error taking screenshot: expected string or bytes-like object
2019-11-12 22:52:36,221 [INFO] - witnessme.py: task_watch - total: 8, done: 4, pending: 4
2019-11-12 22:52:41,221 [INFO] - witnessme.py: task_watch - total: 8, done: 4, pending: 4
2019-11-12 22:52:46,222 [INFO] - witnessme.py: task_watch - total: 8, done: 4, pending: 4
2019-11-12 22:52:51,227 [INFO] - witnessme.py: task_watch - total: 8, done: 4, pending: 4
2019-11-12 22:52:56,230 [INFO] - witnessme.py: task_watch - total: 8, done: 4, pending: 4
2019-11-12 22:53:01,230 [INFO] - witnessme.py: task_watch - total: 8, done: 4, pending: 4
byt3bl33d3r commented 4 years ago

Hmmmm, try replacing line 72 here with this:

screenshot_path = str(pathlib.Path(f'./{report_folder}/{screenshot}').absolute())
ghost commented 4 years ago

That worked! Many thanks!

byt3bl33d3r commented 4 years ago

Awesome. That's actually good to know. It seems that Python interprets pathlib objects differently between Python v3.7.4 and Python >= 3.7.5. I'll commit this change tonight.

ghost commented 4 years ago

Awesome. That's actually good to know. It seems that Python interprets pathlib objects differently between Python v3.7.4 and Python >= 3.7.5. I'll commit this change tonight.

Many thanks! Glad that the issue was informative for you as well.