search

ElevenPaths / thethe

thethe
GNU Affero General Public License v3.0
116 stars 30 forks source link

VirusTotal plugin error when generating MongoDB keys #17

Closed grodriguezl closed 4 years ago

grodriguezl commented 4 years ago

VirusTotal plugin tries to create some MongoDB keys containing a dot on its name like Dr.Web or desenmascara.me. This makes this plugin fail.

Log message for url http://facebook.com/test2 here:

celery_1    | 2020-05-06T20:06:11.171354740Z [2020-05-06 20:06:11,170: WARNING/ForkPoolWorker-1] {'scan_id': 'bb88be418b56dcb1a7d23a8f7b2f3f6c31e70eed5ba1e30988c35bf2d4ab9db4-1433196972', 'resource': 'http://facebook.com/test2', 'url': 'http://facebook.com/test2', 'response_code': 1, 'scan_date': '2015-06-01 22:16:12', 'permalink': 'https://www.virustotal.com/url/bb88be418b56dcb1a7d23a8f7b2f3f6c31e70eed5ba1e30988c35bf2d4ab9db4/analysis/1433196972/', 'verbose_msg': 'Scan finished, scan information embedded in this object', 'filescan_id': None, 'positives': 0, 'total': 63, 'scans': {'CLEAN MX': {'detected': False, 'result': 'clean site'}, 'Spam404': {'detected': False, 'result': 'clean site'}, 'VX Vault': {'detected': False, 'result': 'clean site'}, 'ZDB Zeus': {'detected': False, 'result': 'clean site'}, 'Tencent': {'detected': False, 'result': 'clean site'}, 'MalwarePatrol': {'detected': False, 'result': 'clean site'}, 'ZCloudsec': {'detected': False, 'result': 'clean site'}, 'PhishLabs': {'detected': False, 'result': 'unrated site'}, 'Zerofox': {'detected': False, 'result': 'clean site'}, 'K7AntiVirus': {'detected': False, 'result': 'clean site'}, 'Quttera': {'detected': False, 'result': 'clean site'}, 'AegisLab WebGuard': {'detected': False, 'result': 'clean site'}, 'MalwareDomainList': {'detected': False, 'result': 'clean site', 'detail': 'http://www.malwaredomainlist.com/mdl.php?search=facebook.com'}, 'ZeusTracker': {'detected': False, 'result': 'clean site', 'detail': 'https://zeustracker.abuse.ch/monitor.php?host=facebook.com'}, 'zvelo': {'detected': False, 'result': 'clean site'}, 'Google Safebrowsing': {'detected': False, 'result': 'clean site'}, 'Kaspersky': {'detected': False, 'result': 'clean site'}, 'BitDefender': {'detected': False, 'result': 'clean site'}, 'Dr.Web': {'detected': False, 'result': 'clean site'}, 'ADMINUSLabs': {'detected': False, 'result': 'clean site'}, 'C-SIRT': {'detected': False, 'result': 'clean site'}, 'CyberCrime': {'detected': False, 'result': 'clean site'}, 'Websense ThreatSeeker': {'detected': False, 'result': 'clean site'}, 'CRDF': {'detected': False, 'result': 'clean site'}, 'Webutation': {'detected': False, 'result': 'clean site'}, 'Trustwave': {'detected': False, 'result': 'clean site'}, 'Web Security Guard': {'detected': False, 'result': 'clean site'}, 'G-Data': {'detected': False, 'result': 'clean site'}, 'Malwarebytes hpHosts': {'detected': False, 'result': 'clean site'}, 'Wepawet': {'detected': False, 'result': 'clean site'}, 'AlienVault': {'detected': False, 'result': 'clean site'}, 'Emsisoft': {'detected': False, 'result': 'clean site'}, 'Malc0de Database': {'detected': False, 'result': 'clean site', 'detail': 'http://malc0de.com/database/index.php?search=facebook.com'}, 'SpyEyeTracker': {'detected': False, 'result': 'clean site', 'detail': 'https://spyeyetracker.abuse.ch/monitor.php?host=facebook.com'}, 'malwares.com URL checker': {'detected': False, 'result': 'clean site'}, 'Phishtank': {'detected': False, 'result': 'clean site'}, 'Malwared': {'detected': False, 'result': 'clean site'}, 'Avira': {'detected': False, 'result': 'clean site'}, 'OpenPhish': {'detected': False, 'result': 'clean site'}, 'Antiy-AVL': {'detected': False, 'result': 'clean site'}, 'SCUMWARE.org': {'detected': False, 'result': 'clean site'}, 'FraudSense': {'detected': False, 'result': 'clean site'}, 'Opera': {'detected': False, 'result': 'clean site'}, 'Comodo Site Inspector': {'detected': False, 'result': 'clean site'}, 'Malekal': {'detected': False, 'result': 'clean site'}, 'ESET': {'detected': False, 'result': 'clean site'}, 'Sophos': {'detected': False, 'result': 'unrated site'}, 'Yandex Safebrowsing': {'detected': False, 'result': 'clean site', 'detail': 'http://yandex.com/infected?l10n=en&url=http://facebook.com/test2'}, 'SecureBrain': {'detected': False, 'result': 'clean site'}, 'Malware Domain Blocklist': {'detected': False, 'result': 'clean site'}, 'Blueliv': {'detected': False, 'result': 'clean site'}, 'Netcraft': {'detected': False, 'result': 'unrated site'}, 'PalevoTracker': {'detected': False, 'result': 'clean site'}, 'AutoShun': {'detected': False, 'result': 'unrated site'}, 'ThreatHive': {'detected': False, 'result': 'clean site'}, 'ParetoLogic': {'detected': False, 'result': 'clean site'}, 'Rising': {'detected': False, 'result': 'clean site'}, 'URLQuery': {'detected': False, 'result': 'unrated site'}, 'StopBadware': {'detected': False, 'result': 'unrated site'}, 'Sucuri SiteCheck': {'detected': False, 'result': 'clean site'}, 'Fortinet': {'detected': False, 'result': 'clean site'}, 'ZeroCERT': {'detected': False, 'result': 'clean site'}, 'Baidu-International': {'detected': False, 'result': 'clean site'}}}
celery_1    | 2020-05-06T20:06:11.216380190Z [2020-05-06 20:06:11,216: WARNING/ForkPoolWorker-1] Traceback (most recent call last):
celery_1    | 2020-05-06T20:06:11.216428577Z   File "/usr/src/thethe/server/plugins/virustotal.py", line 113, in virustotal
celery_1    | 2020-05-06T20:06:11.216436078Z     resource_id, plugin_name, project_id, response, result_status
celery_1    | 2020-05-06T20:06:11.216441855Z   File "/usr/src/thethe/server/entities/plugin_manager.py", line 235, in set_plugin_results
celery_1    | 2020-05-06T20:06:11.216447390Z     "result_status": result_status.value,
celery_1    | 2020-05-06T20:06:11.216451924Z   File "/usr/local/lib/python3.7/site-packages/pymongo/collection.py", line 698, in insert_one
celery_1    | 2020-05-06T20:06:11.216456437Z     session=session),
celery_1    | 2020-05-06T20:06:11.216460640Z   File "/usr/local/lib/python3.7/site-packages/pymongo/collection.py", line 612, in _insert
celery_1    | 2020-05-06T20:06:11.216465037Z     bypass_doc_val, session)
celery_1    | 2020-05-06T20:06:11.216469462Z   File "/usr/local/lib/python3.7/site-packages/pymongo/collection.py", line 600, in _insert_one
celery_1    | 2020-05-06T20:06:11.216474099Z     acknowledged, _insert_command, session)
celery_1    | 2020-05-06T20:06:11.216478257Z   File "/usr/local/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1492, in _retryable_write
celery_1    | 2020-05-06T20:06:11.216482749Z     return self._retry_with_session(retryable, func, s, None)
celery_1    | 2020-05-06T20:06:11.216487647Z   File "/usr/local/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1385, in _retry_with_session
celery_1    | 2020-05-06T20:06:11.216492568Z     return func(session, sock_info, retryable)
celery_1    | 2020-05-06T20:06:11.216496695Z   File "/usr/local/lib/python3.7/site-packages/pymongo/collection.py", line 595, in _insert_command
celery_1    | 2020-05-06T20:06:11.216501149Z     retryable_write=retryable_write)
celery_1    | 2020-05-06T20:06:11.216505300Z   File "/usr/local/lib/python3.7/site-packages/pymongo/pool.py", line 618, in command
celery_1    | 2020-05-06T20:06:11.216509807Z     self._raise_connection_failure(error)
celery_1    | 2020-05-06T20:06:11.216513835Z   File "/usr/local/lib/python3.7/site-packages/pymongo/pool.py", line 613, in command
celery_1    | 2020-05-06T20:06:11.216518167Z     user_fields=user_fields)
celery_1    | 2020-05-06T20:06:11.216522107Z   File "/usr/local/lib/python3.7/site-packages/pymongo/network.py", line 129, in command
celery_1    | 2020-05-06T20:06:11.216526354Z     codec_options, ctx=compression_ctx)
celery_1    | 2020-05-06T20:06:11.216530394Z   File "/usr/local/lib/python3.7/site-packages/pymongo/message.py", line 704, in _op_msg
celery_1    | 2020-05-06T20:06:11.216534732Z     flags, command, identifier, docs, check_keys, opts)
celery_1    | 2020-05-06T20:06:11.216538759Z bson.errors.InvalidDocument: key 'Dr.Web' must not contain '.'
deibit commented 4 years ago

Hi Guillermo.

Thanks for reporting this. We are investigating the issue.

Regards.

deibit commented 4 years ago

Should be fixed in https://github.com/ElevenPaths/thethe/commit/33cb691a6d42a51f194e358f0046ad3082bede66