Write crawler targeting mode

gingeleski commented 5 years ago

Write crawler targeting mode

Conspiracy crawler

load the page
isolate clickable elements that are both visible to the naked eye and "do something"
any invisible clickable elements should be stored in case they are of value later, maybe just to call attention to them for manual pentester

gingeleski commented 5 years ago

Can take inspiration from https://github.com/yujiosaka/headless-chrome-crawler

gingeleski commented 5 years ago

Getting the following error when trying a test run on my personal site...

(venv) PS C:\Users\gingeleski\workspace\conspiracy> python conspiracy.py --targeting-mode crawl gingeleski.com
2019-05-14 23:12:58,373 [INFO]
2019-05-14 23:12:58,374 [INFO]  ######   #######  ##    ##  ######  ########  #### ########     ###     ######  ##    ##
2019-05-14 23:12:58,382 [INFO] ##    ## ##     ## ###   ## ##    ## ##     ##  ##  ##     ##   ## ##   ##    ##  ##  ##
2019-05-14 23:12:58,386 [INFO] ##       ##     ## ####  ## ##       ##     ##  ##  ##     ##  ##   ##  ##         ####
2019-05-14 23:12:58,392 [INFO] ##       ##     ## ## ## ##  ######  ########   ##  ########  ##     ## ##          ##
2019-05-14 23:12:58,393 [INFO] ##       ##     ## ##  ####       ## ##         ##  ##   ##   ######### ##          ##
2019-05-14 23:12:58,417 [INFO] ##    ## ##     ## ##   ### ##    ## ##         ##  ##    ##  ##     ## ##    ##    ##
2019-05-14 23:12:58,417 [INFO]  ######   #######  ##    ##  ######  ##        #### ##     ## ##     ##  ######     ##
2019-05-14 23:12:58,418 [INFO]
2019-05-14 23:12:58,418 [INFO] Conspiracy v0.1 - Automated web app hacking
2019-05-14 23:12:58,418 [INFO]
2019-05-14 23:12:58,418 [INFO] Starting to parse given targets...
2019-05-14 23:12:58,418 [INFO] Looking for match to given targeting mode string "crawl"
2019-05-14 23:12:58,419 [INFO] Matched targeting mode "Crawl"
2019-05-14 23:12:58,419 [INFO] Conducting Crawl targeting now...
[I:pyppeteer.launcher] Browser listening on: ws://127.0.0.1:5856/devtools/browser/a75d1a9c-508a-4949-9b2d-67ff826ebc45
Consumer 0 Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?
[I:pyppeteer.connection] connection closed
2019-05-14 23:13:19,063 [ERROR] Future exception was never retrieved
future: <Future finished exception=ConnectionClosed('WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason
',)>
Traceback (most recent call last):
  File "C:\Users\gingeleski\workspace\conspiracy\venv\lib\site-packages\websockets\protocol.py", line 674, in transfer_data
    message = yield from self.read_message()
  File "C:\Users\gingeleski\workspace\conspiracy\venv\lib\site-packages\websockets\protocol.py", line 742, in read_message
    frame = yield from self.read_data_frame(max_size=self.max_size)
  File "C:\Users\gingeleski\workspace\conspiracy\venv\lib\site-packages\websockets\protocol.py", line 815, in read_data_frame
    frame = yield from self.read_frame(max_size)
  File "C:\Users\gingeleski\workspace\conspiracy\venv\lib\site-packages\websockets\protocol.py", line 884, in read_frame
    extensions=self.extensions,
  File "C:\Users\gingeleski\workspace\conspiracy\venv\lib\site-packages\websockets\framing.py", line 99, in read
    data = yield from reader(2)
  File "c:\users\gingeleski\anaconda3\Lib\asyncio\streams.py", line 666, in readexactly
    raise IncompleteReadError(incomplete, n)
asyncio.streams.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason

gingeleski / conspiracy

Write crawler targeting mode #42

Conspiracy crawler