NoSuchTableException @ grep plugins

1d3df9903ad commented 9 years ago

History

[x] Bugs reported by automated exception handler
[x] Tried to reproduce and failed
[x] Tried to identify the bug in the source code and failed
[x] Added debug information https://github.com/andresriancho/w3af/commit/07acc19cc015cef5f753e16b9728156cad9e5b8b
[ ] Waiting for new bug reports containing the debug information
[ ] Fix
Analysis

These are the plugins where this error has been found in the past:

html_comments
audit.ssi

So I'm going to add some debug code to the plugins to better understand it's internal state and wait for further bug reports.

Incorrect theory number: 1

The scan ends, we call "clear" on the grep plugin, but a thread is still running and sends a request which gets analyzed by the grep plugin.

We might fix this by doing better thread stop, not clearing grep, etc.

I've analyze the code and this doesn't seem to be possible due to the grep inqueue and the way things are handled.

Version Information

  Python version: 2.7.3 (default, Mar 13 2014, 11:03:55) [GCC 4.7.2]
  GTK version: 2.24.10
  PyGTK version: 2.24.0
  w3af version:
    w3af - Web Application Attack and Audit Framework
    Version: 1.6.54
    Distribution: Kali Linux
    Author: Andres Riancho and the w3af team.

Traceback

A "NoSuchTableException" exception was found while running grep.html_comments on "Method: GET | http://domain/". The exception was: "no such table: disk_dict_html_comments_FvfNscLmHXcIhvxiKkJFtScGzdkecs" at _base.py:__get_result():357.The full traceback is:
  File "/usr/share/w3af/w3af/core/controllers/core_helpers/consumers/grep.py", line 100, in _consume
    plugin.grep_wrapper(request, response)
  File "/usr/share/w3af/w3af/core/controllers/plugins/grep_plugin.py", line 55, in grep_wrapper
    self.grep(fuzzable_request, response)
  File "/usr/share/w3af/w3af/plugins/grep/html_comments.py", line 94, in grep
    if self._is_new(comment, response):
  File "/usr/share/w3af/w3af/plugins/grep/html_comments.py", line 161, in _is_new
    comment_data = self._comments.get(comment, None)
  File "/usr/share/w3af/w3af/core/data/db/disk_dict.py", line 129, in get
    if key in self:
  File "/usr/share/w3af/w3af/core/data/db/disk_dict.py", line 90, in __contains__
    r = self.db.select_one(query, (cpickle_dumps(key),))
  File "/usr/share/w3af/w3af/core/data/db/dbms.py", line 69, in inner_verify_started
    return meth(self, *args, **kwds)
  File "/usr/share/w3af/w3af/core/data/db/dbms.py", line 137, in select_one
    return self.select(query, parameters)[0]
  File "/usr/share/w3af/w3af/core/data/db/dbms.py", line 69, in inner_verify_started
    return meth(self, *args, **kwds)
  File "/usr/share/w3af/w3af/core/data/db/dbms.py", line 128, in select
    return future.result()
  File "/usr/lib/pymodules/python2.7/concurrent/futures/_base.py", line 405, in result
    return self.__get_result()
  File "/usr/lib/pymodules/python2.7/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception

Extra debugging information

https://github.com/andresriancho/w3af/issues/14599

A "NoSuchTableException" exception was found while running grep.html_comments on "Method: GET | https://domain/". The exception was: "A NoSuchTableException was raised by the DBMS. This issue is related with #10849 , but since I was unable to reproduce it, extra debug information is added to the exception:

 - Grep plugin end() was called: True
 - Response ID is: 43
 - HTML comment is: " Investments Menu "
 - Original exception: "no such table: disk_dict_html_comments_klUMqpnrxJkgfLicrRpGQApoilSzBX"

lreading commented 6 years ago

If it helps at all, I'm getting this exception with the web_spider plugin. Please let me know if any kind of logs would be helpful for you.

The plugin "web_spider" raised an exception in the end() method: no such table: disk_list_web_spider_EehHHFbBjQbrlrEwFWWfMzoBqZTxEL
The following error was detected and could not be resolved:
Failed to initialize the 404 detection, original exception was: "no such table: disk_dict_deque_pbJyKDLXgNxUnhetrqNomjJxOJEomL".

andresriancho commented 6 years ago

@lreading thanks for the comment. You get this exception each time you run a scan? I always thought about this issue as a race condition that happens 1/100 scans, thus never paid attention to it.

Let me know and we'll see if we can fix it.

andresriancho / w3af