CIRCL / AIL-framework

AIL framework - Analysis Information Leak framework. Project moved to https://github.com/ail-project
https://github.com/ail-project/ail-framework
GNU Affero General Public License v3.0
1.3k stars 282 forks source link

Crash on global search #397

Closed ghost closed 7 months ago

ghost commented 4 years ago

During the Prague training I encountered Internal Server Error whenever I tried to use the global search (no matter the input). Backtrace: 172.20.7.133 - - [19/Sep/2019 15:33:41] "POST /search HTTP/1.1" 500 - [2019-09-19 15:34:00,187] ERROR in app: Exception on /search [POST] Traceback (most recent call last): File "/home/ail/AIL-framework/AILENV/lib/python3.5/site-packages/flask/app.py", line 2446, in wsgi_app response = self.full_dispatch_request() File "/home/ail/AIL-framework/AILENV/lib/python3.5/site-packages/flask/app.py", line 1951, in full_dispatch_request rv = self.handle_user_exception(e) File "/home/ail/AIL-framework/AILENV/lib/python3.5/site-packages/flask/app.py", line 1820, in handle_user_exception reraise(exc_type, exc_value, tb) File "/home/ail/AIL-framework/AILENV/lib/python3.5/site-packages/flask/_compat.py", line 39, in reraise raise value File "/home/ail/AIL-framework/AILENV/lib/python3.5/site-packages/flask/app.py", line 1949, in full_dispatch_request rv = self.dispatch_request() File "/home/ail/AIL-framework/AILENV/lib/python3.5/site-packages/flask/app.py", line 1935, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/home/ail/AIL-framework/AILENV/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view return func(*args, **kwargs) File "./modules/Role_Manager.py", line 60, in decorated_view return func(*args, **kwargs) File "/home/ail/AIL-framework/var/www/modules/search/Flask_search.py", line 136, in search ix = index.open_dir(selected_index) File "/home/ail/AIL-framework/AILENV/lib/python3.5/site-packages/whoosh/index.py", line 123, in open_dir return FileIndex(storage, schema=schema, indexname=indexname) File "/home/ail/AIL-framework/AILENV/lib/python3.5/site-packages/whoosh/index.py", line 421, in __init__ TOC.read(self.storage, self.indexname, schema=self._schema) File "/home/ail/AIL-framework/AILENV/lib/python3.5/site-packages/whoosh/index.py", line 632, in read check_size("int", _INT_SIZE) File "/home/ail/AIL-framework/AILENV/lib/python3.5/site-packages/whoosh/index.py", line 626, in check_size sz = stream.read_varint() File "/home/ail/AIL-framework/AILENV/lib/python3.5/site-packages/whoosh/filedb/structfile.py", line 191, in read_varint return read_varint(self.read) File "/home/ail/AIL-framework/AILENV/lib/python3.5/site-packages/whoosh/util/varints.py", line 102, in read_varint b = ord(readfn(1))

Terrtia commented 4 years ago

This error is caused by a whoosh index corruption : https://stackoverflow.com/questions/7949555/strange-error-adding-to-whoosh-index

It can happen when the module is killed when writing on the index

Deleting all files in /indexdir solve this issue

195

Anderson0abr commented 4 years ago

Hi @Terrtia

Deleting the /indexdir fixed the issue, but of course we lose all the info indexed until now..

Is there a way to reindex the data?

Terrtia commented 4 years ago

Hi @Anderson0abr !

You need to reprocess all Pastes/Items with the Indexer module.

We plan to add a way to reprocess Items by modules/daterange. #275