search

matamorphosis / Scrummage

A Holistic OSINT and Threat Hunting Platform
GNU General Public License v3.0
502 stars 81 forks source link

Google searches failing when scheduled #17

Closed Joukahainen123 closed 3 years ago

Joukahainen123 commented 3 years ago

Hi

When I create a Google search and run it, it works fine. But when I schedule an task, or run the task with /usr/bin/python3 /home/username/Scrummage/lib/plugin_caller.py -t 1

I get the following error, and new results are not found:

WARNING - 2020-10-18 11:49:37 - Google_Search - HTTPSConnectionPool(host='www.myservernamewashere.net, port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))) WARNING - 2020-10-18 11:49:37 - Google_Search - HTTPSConnectionPool(host='www.myservernamewashere.net', port=443): Max retries exceeded with url: /?p=4 (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))) WARNING - 2020-10-18 11:49:37 - Google_Search - Failed to match regular expression. WARNING - 2020-10-18 11:49:37 - Google_Search - Failed to match regular expression. INFO - 2020-10-18 11:49:37 Connectors Library - Loading Scrummage's Main Database configuration data.

matamorphosis commented 3 years ago

Hi Joukahainen123,

Not sure why you are getting this issue. When I run a create a Google Search Task with a random query like "facebook", directly from the plugin caller, I get:

me@linux:/path/to/Scrummage/lib$ python3 plugin_caller.py -t 39
INFO:root:2020-10-28 23:50:16 General Library - Using directory: /path/to/Scrummage/lib/static/protected/output/google/2020/10/28.
INFO:root:2020-10-28 23:50:16 - Google_Search - Loading configuration data.
INFO:googleapiclient.discovery:URL being requested: GET https://www.googleapis.com/discovery/v1/apis/customsearch/v1/rest?key=AIzaSyDP2RzMOu13-pr-rp2y3jtl-0VSnlc1DRo
INFO:googleapiclient.discovery:URL being requested: GET https://customsearch.googleapis.com/customsearch/v1?q=facebook&cx=006462069640390212720%3A3m0uxjec2no&start=1&num=10&key=AIzaSyDP2RzMOu13-pr-rp2y3jtl-0VSnlc1DRo&alt=json
INFO:root:2020-10-28 23:50:17 General Library - Main file created.
INFO:root:2020-10-28 23:50:24 General Library - File: /path/to/Scrummage/lib/static/protected/output/google/2020/10/28/Google-Query-facebook-1-appserve-mkt-p-AFnwnKXfd0Q13phvl5g2cXtkAQeWo07hmv4gDNb-T_IvgpO5a3VbUDsRQWelMxyUaz668V3lrY8l-cNRmPN6bCLLAvTxTLTETe1tVDrWdASt.html created.
INFO:root:2020-10-28 23:50:24 General Library - Adding item to Scrummage database and other configured outputs.
INFO:root:2020-10-28 23:50:24 Connectors Library - Loading CSV configuration data.
INFO:root:2020-10-28 23:50:24 Connectors Library - Updated existing CSV file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.csv.
INFO:root:2020-10-28 23:50:24 Connectors Library - Loading DOCX configuration data.
INFO:root:2020-10-28 23:50:24 Connectors Library - Exported to DOCX file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.docx.
INFO:root:2020-10-28 23:50:24 Connectors Library - Loading Scrummage's Main Database configuration data.
INFO:root:2020-10-28 23:50:24 Connectors Library - Loading Elasticsearch configuration data.
INFO:root:2020-10-28 23:50:24 Connectors Library - Loading DefectDojo configuration data.
INFO:root:2020-10-28 23:50:24 Connectors Library - Loading Scumblr configuration data.
INFO:root:2020-10-28 23:50:24 Connectors Library - Loading RTIR configuration data.
INFO:root:2020-10-28 23:50:24 Connectors Library - Loading JIRA configuration data.
INFO:root:2020-10-28 23:50:24 Connectors Library - Loading email configuration data.
INFO:root:2020-10-28 23:50:24 Connectors Library - Loading Slack configuration data.
INFO:root:2020-10-28 23:50:27 General Library - File: /path/to/Scrummage/lib/static/protected/output/google/2020/10/28/Google-Query-facebook-1-appserve-mkt-p-AL8lKjPoawP9LUY6j-mHjE2ss4OZwLAdwo5GJeg3D037HQNMuSG9XNWNEjF6VPLwDuq_PDdSVqExEmBPDw1nFHr9Q4Jn-PE93ySF6TLavIqId8hd_LMEsA.html created.
INFO:root:2020-10-28 23:50:27 General Library - Adding item to Scrummage database and other configured outputs.
INFO:root:2020-10-28 23:50:27 Connectors Library - Loading CSV configuration data.
INFO:root:2020-10-28 23:50:27 Connectors Library - Updated existing CSV file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.csv.
INFO:root:2020-10-28 23:50:27 Connectors Library - Loading DOCX configuration data.
INFO:root:2020-10-28 23:50:27 Connectors Library - Exported to DOCX file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.docx.
INFO:root:2020-10-28 23:50:27 Connectors Library - Loading Scrummage's Main Database configuration data.
INFO:root:2020-10-28 23:50:27 Connectors Library - Loading Elasticsearch configuration data.
INFO:root:2020-10-28 23:50:27 Connectors Library - Loading DefectDojo configuration data.
INFO:root:2020-10-28 23:50:27 Connectors Library - Loading Scumblr configuration data.
INFO:root:2020-10-28 23:50:27 Connectors Library - Loading RTIR configuration data.
INFO:root:2020-10-28 23:50:27 Connectors Library - Loading JIRA configuration data.
INFO:root:2020-10-28 23:50:27 Connectors Library - Loading email configuration data.
INFO:root:2020-10-28 23:50:27 Connectors Library - Loading Slack configuration data.
INFO:root:2020-10-28 23:50:30 General Library - File: /path/to/Scrummage/lib/static/protected/output/google/2020/10/28/Google-Query-facebook-1-appserve-mkt-p-AIQrb_5-rfli-lyK6O9VA3GzzmQhJkhmGO7LLpiRBwsuZUdkeJB7HdNfCH_nPPZc1I2mEki_H3vnJRSGu-Q5HnE9CqXsVuefm3w.html created.
INFO:root:2020-10-28 23:50:30 General Library - Adding item to Scrummage database and other configured outputs.
INFO:root:2020-10-28 23:50:30 Connectors Library - Loading CSV configuration data.
INFO:root:2020-10-28 23:50:30 Connectors Library - Updated existing CSV file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.csv.
INFO:root:2020-10-28 23:50:30 Connectors Library - Loading DOCX configuration data.
INFO:root:2020-10-28 23:50:30 Connectors Library - Exported to DOCX file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.docx.
INFO:root:2020-10-28 23:50:30 Connectors Library - Loading Scrummage's Main Database configuration data.
INFO:root:2020-10-28 23:50:30 Connectors Library - Loading Elasticsearch configuration data.
INFO:root:2020-10-28 23:50:30 Connectors Library - Loading DefectDojo configuration data.
INFO:root:2020-10-28 23:50:30 Connectors Library - Loading Scumblr configuration data.
INFO:root:2020-10-28 23:50:30 Connectors Library - Loading RTIR configuration data.
INFO:root:2020-10-28 23:50:30 Connectors Library - Loading JIRA configuration data.
INFO:root:2020-10-28 23:50:30 Connectors Library - Loading email configuration data.
INFO:root:2020-10-28 23:50:30 Connectors Library - Loading Slack configuration data.
INFO:root:2020-10-28 23:50:33 General Library - File: /path/to/Scrummage/lib/static/protected/output/google/2020/10/28/Google-Query-facebook-1-appserve-mkt-p-AM7kBiUhEUCzmhZFNvDHzHyWMkJg6XXmViBUZg4-aNLn-9h-jWOhNmECkd6Ht25b3JamxVGsoB56ZqAHE4VL6ZzPSrm3geaG9yNyz_fcVPQ9wf2bgoH-NWWG.html created.
INFO:root:2020-10-28 23:50:33 General Library - Adding item to Scrummage database and other configured outputs.
INFO:root:2020-10-28 23:50:33 Connectors Library - Loading CSV configuration data.
INFO:root:2020-10-28 23:50:33 Connectors Library - Updated existing CSV file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.csv.
INFO:root:2020-10-28 23:50:33 Connectors Library - Loading DOCX configuration data.
INFO:root:2020-10-28 23:50:33 Connectors Library - Exported to DOCX file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.docx.
INFO:root:2020-10-28 23:50:33 Connectors Library - Loading Scrummage's Main Database configuration data.
INFO:root:2020-10-28 23:50:33 Connectors Library - Loading Elasticsearch configuration data.
INFO:root:2020-10-28 23:50:33 Connectors Library - Loading DefectDojo configuration data.
INFO:root:2020-10-28 23:50:33 Connectors Library - Loading Scumblr configuration data.
INFO:root:2020-10-28 23:50:33 Connectors Library - Loading RTIR configuration data.
INFO:root:2020-10-28 23:50:33 Connectors Library - Loading JIRA configuration data.
INFO:root:2020-10-28 23:50:33 Connectors Library - Loading email configuration data.
INFO:root:2020-10-28 23:50:33 Connectors Library - Loading Slack configuration data.
WARNING:root:2020-10-28 23:50:34 General Library - Failed to create query file.
WARNING:root:2020-10-28 23:50:34 - Google_Search - Failed to create output file. File may already exist.
INFO:root:2020-10-28 23:50:37 General Library - File: /path/to/Scrummage/lib/static/protected/output/google/2020/10/28/Google-Query-facebook-1-appserve-mkt-p-AJ-PF7wqu8u4dPrGuI4ECr-rGCyV94Ufny3W6WLBAq2PJEdxI100UzkIN7JWoPqq4o8B2obdztOM6l8tyT9jutlTy_tTfU_erJf_uupgD7Z816WzOaYtDKuOXwIXyekWbILzrCgzZTIdTDekXPyJQMMBzw.html created.
INFO:root:2020-10-28 23:50:37 General Library - Adding item to Scrummage database and other configured outputs.
INFO:root:2020-10-28 23:50:37 Connectors Library - Loading CSV configuration data.
INFO:root:2020-10-28 23:50:37 Connectors Library - Updated existing CSV file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.csv.
INFO:root:2020-10-28 23:50:37 Connectors Library - Loading DOCX configuration data.
INFO:root:2020-10-28 23:50:37 Connectors Library - Exported to DOCX file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.docx.
INFO:root:2020-10-28 23:50:37 Connectors Library - Loading Scrummage's Main Database configuration data.
INFO:root:2020-10-28 23:50:37 Connectors Library - Loading Elasticsearch configuration data.
INFO:root:2020-10-28 23:50:37 Connectors Library - Loading DefectDojo configuration data.
INFO:root:2020-10-28 23:50:37 Connectors Library - Loading Scumblr configuration data.
INFO:root:2020-10-28 23:50:37 Connectors Library - Loading RTIR configuration data.
INFO:root:2020-10-28 23:50:37 Connectors Library - Loading JIRA configuration data.
INFO:root:2020-10-28 23:50:37 Connectors Library - Loading email configuration data.
INFO:root:2020-10-28 23:50:37 Connectors Library - Loading Slack configuration data.
INFO:root:2020-10-28 23:50:39 General Library - File: /path/to/Scrummage/lib/static/protected/output/google/2020/10/28/Google-Query-facebook-1-appserve-mkt-p-AFnwnKVxWjoE7Q59xkQNUON6_8aTuEB3GK5TH0QsTBPxgXS2vfWI7mLvV9UrlGeRmNht81LKmwTVeovqp5G5-J2pXkRRlHnVVpL87l0_KyeVq37GeBbfP8oYFGVQ9Q.html created.
INFO:root:2020-10-28 23:50:39 General Library - Adding item to Scrummage database and other configured outputs.
INFO:root:2020-10-28 23:50:39 Connectors Library - Loading CSV configuration data.
INFO:root:2020-10-28 23:50:39 Connectors Library - Updated existing CSV file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.csv.
INFO:root:2020-10-28 23:50:39 Connectors Library - Loading DOCX configuration data.
INFO:root:2020-10-28 23:50:39 Connectors Library - Exported to DOCX file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.docx.
INFO:root:2020-10-28 23:50:39 Connectors Library - Loading Scrummage's Main Database configuration data.
INFO:root:2020-10-28 23:50:39 Connectors Library - Loading Elasticsearch configuration data.
INFO:root:2020-10-28 23:50:39 Connectors Library - Loading DefectDojo configuration data.
INFO:root:2020-10-28 23:50:39 Connectors Library - Loading Scumblr configuration data.
INFO:root:2020-10-28 23:50:39 Connectors Library - Loading RTIR configuration data.
INFO:root:2020-10-28 23:50:39 Connectors Library - Loading JIRA configuration data.
INFO:root:2020-10-28 23:50:39 Connectors Library - Loading email configuration data.
INFO:root:2020-10-28 23:50:39 Connectors Library - Loading Slack configuration data.
INFO:root:2020-10-28 23:50:45 General Library - File: /path/to/Scrummage/lib/static/protected/output/google/2020/10/28/Google-Query-facebook-1-appserve-mkt-p-AD-FnEy568fGSEj5dtRDafWYluzzFqp3wTqnCyXZaJ1eYv4B4lMuapAQKVpnrgJgtQkxAUm0L438kGDOscWNM1vKj2TuGljAadN7TVkZI6qG2VM7u8-BbvXFnTi9OMizwAqMtw.html created.
INFO:root:2020-10-28 23:50:45 General Library - Adding item to Scrummage database and other configured outputs.
INFO:root:2020-10-28 23:50:45 Connectors Library - Loading CSV configuration data.
INFO:root:2020-10-28 23:50:45 Connectors Library - Updated existing CSV file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.csv.
INFO:root:2020-10-28 23:50:45 Connectors Library - Loading DOCX configuration data.
INFO:root:2020-10-28 23:50:45 Connectors Library - Exported to DOCX file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.docx.
INFO:root:2020-10-28 23:50:45 Connectors Library - Loading Scrummage's Main Database configuration data.
INFO:root:2020-10-28 23:50:45 Connectors Library - Loading Elasticsearch configuration data.
INFO:root:2020-10-28 23:50:45 Connectors Library - Loading DefectDojo configuration data.
INFO:root:2020-10-28 23:50:45 Connectors Library - Loading Scumblr configuration data.
INFO:root:2020-10-28 23:50:45 Connectors Library - Loading RTIR configuration data.
INFO:root:2020-10-28 23:50:45 Connectors Library - Loading JIRA configuration data.
INFO:root:2020-10-28 23:50:45 Connectors Library - Loading email configuration data.
INFO:root:2020-10-28 23:50:45 Connectors Library - Loading Slack configuration data.
INFO:googleapiclient.discovery:URL being requested: GET https://www.googleapis.com/discovery/v1/apis/customsearch/v1/rest?key=AIzaSyDP2RzMOu13-pr-rp2y3jtl-0VSnlc1DRo
INFO:googleapiclient.discovery:URL being requested: GET https://customsearch.googleapis.com/customsearch/v1?q=facebook&cx=006462069640390212720%3A3m0uxjec2no&start=11&num=10&key=AIzaSyDP2RzMOu13-pr-rp2y3jtl-0VSnlc1DRo&alt=json
INFO:root:2020-10-28 23:50:46 General Library - Main file created.
INFO:root:2020-10-28 23:50:47 General Library - File: /path/to/Scrummage/lib/static/protected/output/google/2020/10/28/Google-Query-facebook-11-finance.html created.
INFO:root:2020-10-28 23:50:47 General Library - Adding item to Scrummage database and other configured outputs.
INFO:root:2020-10-28 23:50:47 Connectors Library - Loading CSV configuration data.
INFO:root:2020-10-28 23:50:47 Connectors Library - Updated existing CSV file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.csv.
INFO:root:2020-10-28 23:50:47 Connectors Library - Loading DOCX configuration data.
INFO:root:2020-10-28 23:50:47 Connectors Library - Exported to DOCX file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.docx.
INFO:root:2020-10-28 23:50:47 Connectors Library - Loading Scrummage's Main Database configuration data.
INFO:root:2020-10-28 23:50:47 Connectors Library - Loading Elasticsearch configuration data.
INFO:root:2020-10-28 23:50:47 Connectors Library - Loading DefectDojo configuration data.
INFO:root:2020-10-28 23:50:47 Connectors Library - Loading Scumblr configuration data.
INFO:root:2020-10-28 23:50:47 Connectors Library - Loading RTIR configuration data.
INFO:root:2020-10-28 23:50:47 Connectors Library - Loading JIRA configuration data.
INFO:root:2020-10-28 23:50:47 Connectors Library - Loading email configuration data.
INFO:root:2020-10-28 23:50:47 Connectors Library - Loading Slack configuration data.
INFO:root:2020-10-28 23:50:50 General Library - File: /path/to/Scrummage/lib/static/protected/output/google/2020/10/28/Google-Query-facebook-11-finance-quote-FB.html created.
INFO:root:2020-10-28 23:50:50 General Library - Adding item to Scrummage database and other configured outputs.
INFO:root:2020-10-28 23:50:50 Connectors Library - Loading CSV configuration data.
INFO:root:2020-10-28 23:50:50 Connectors Library - Updated existing CSV file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.csv.
INFO:root:2020-10-28 23:50:50 Connectors Library - Loading DOCX configuration data.
INFO:root:2020-10-28 23:50:50 Connectors Library - Exported to DOCX file located at /path/to/Scrummage/lib/static/protected/output/google/Google-Output.docx.
INFO:root:2020-10-28 23:50:50 Connectors Library - Loading Scrummage's Main Database configuration data.
INFO:root:2020-10-28 23:50:50 Connectors Library - Loading Elasticsearch configuration data.
INFO:root:2020-10-28 23:50:50 Connectors Library - Loading DefectDojo configuration data.
INFO:root:2020-10-28 23:50:50 Connectors Library - Loading Scumblr configuration data.
INFO:root:2020-10-28 23:50:50 Connectors Library - Loading RTIR configuration data.
INFO:root:2020-10-28 23:50:50 Connectors Library - Loading JIRA configuration data.
INFO:root:2020-10-28 23:50:50 Connectors Library - Loading email configuration data.
INFO:root:2020-10-28 23:50:50 Connectors Library - Loading Slack configuration data.

Some basic things to check:

When I Google the error "_(Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_servercertificate', 'certificate verify failed')])")))" the recommended solution is to disable SSL verification. If it isn't too much trouble could I get you to replace the following text on line 99 of the Google_Search.py plugin:

Google_Item_Response = requests.get(Google_Item_URL, headers=General.URL_Headers(User_Agent=True, Application_JSON_CT=True, Accept_XML=True, Accept_Language_EN_US=True)).text

with the following:

Google_Item_Response = requests.get(Google_Item_URL, headers=General.URL_Headers(User_Agent=True, Application_JSON_CT=True, Accept_XML=True, Accept_Language_EN_US=True), verify=False).text

Then rerun plugin_caller.py against your task. If this resolves your issue I will think of a way I can fix this across all plugins.

Regards,

Matamorphosis

Joukahainen123 commented 3 years ago

I am running 2.6. Will first upgrade to 2.7.

Joukahainen123 commented 3 years ago

Did upgrade to 2.7 and changed the Google_Search.py as instructed above.

Strange thing here is that when I create the task and run it the first time everything works. Bun if I try to rerun the same task, from console, or from GUI it fails cant see the Connectors Library for example).

INFO:root:2020-10-28 17:53:06 General Library - File: /home/mehmeies/Scrummage/lib/static/protected/output/google/2020/10/28/Google-Query-TEST-1-people-Sulo-V.html created. INFO:root:2020-10-28 17:53:06 General Library - Adding item to Scrummage database and other configured outputs. INFO:root:2020-10-28 17:53:06 Connectors Library - Loading CSV configuration data. INFO:root:2020-10-28 17:53:06 Connectors Library - Loading DOCX configuration data. INFO:root:2020-10-28 17:53:06 Connectors Library - Loading Scrummage's Main Database configuration data. INFO:root:2020-10-28 17:53:06 Connectors Library - Loading Elasticsearch configuration data. INFO:root:2020-10-28 17:53:06 Connectors Library - Loading DefectDojo configuration data. INFO:root:2020-10-28 17:53:06 Connectors Library - Loading Scumblr configuration data. INFO:root:2020-10-28 17:53:06 Connectors Library - Loading RTIR configuration data. INFO:root:2020-10-28 17:53:06 Connectors Library - Loading JIRA configuration data. INFO:root:2020-10-28 17:53:06 Connectors Library - Loading email configuration data. INFO:root:2020-10-28 17:53:07 Connectors Library - Email Sent. INFO:root:2020-10-28 17:53:07 Connectors Library - Loading Slack configuration data. INFO:root:2020-10-28 17:54:11 - Google_Search - Loading configuration data. INFO:werkzeug:192.168.1.105 - - [28/Oct/2020 17:54:11] "POST /tasks/run/1 HTTP/1.1" 302 - WARNING:root:2020-10-28 17:54:11 General Library - Using directory: /home/mehmeies/Scrummage/lib/static/protected/output/google/2020/10/28. INFO:root:2020-10-28 17:54:11 - Google_Search - Loading configuration data. INFO:werkzeug:192.168.1.105 - - [28/Oct/2020 17:54:11] "GET /tasks HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [28/Oct/2020 17:54:11] "GET /static/css/template.css HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [28/Oct/2020 17:54:11] "GET /static/js/main.js HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [28/Oct/2020 17:54:11] "GET /static/js/jquery-3.4.1.min.js HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [28/Oct/2020 17:54:11] "GET /static/js/datatables.min.js HTTP/1.1" 200 - INFO:root:2020-10-28 17:54:12 General Library - Main file created. /usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host 'servernamewashere'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings warnings.warn( WARNING:root:2020-10-28 17:54:12 - Google_Search - 'NoneType' object has no attribute 'replace' /usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host 'servernamewashere'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings warnings.warn( WARNING:root:2020-10-28 17:54:12 - Google_Search - 'NoneType' object has no attribute 'replace' WARNING:root:2020-10-28 17:54:12 - Google_Search - Failed to match regular expression. INFO:werkzeug:192.168.1.105 - - [28/Oct/2020 17:54:12] "GET /static/images/search_grey.png HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [28/Oct/2020 17:54:12] "GET /static/images/sort_asc.png HTTP/1.1" 200 -

matamorphosis commented 3 years ago

So I did identify a flawed assumption in the regex logic, I have improved this so it should fix the issues such as 'NoneType' object has no attribute 'replace'. I have updated the plugin in GitHub appropriately. Let me know if this fixes your other issues, which I'm not saying it necessarily will.

As for the "Failed to match regular expression", could you please add print(Google_Item_URL) to the Google_Search.py plugin, underneath line 126 (logging.warning(f"{General.Date()} - {name.strip('plugins.')} - Failed to match regular expression.")) s this will help me identify is it is an issue with the data or my regex.

Cheers

Joukahainen123 commented 3 years ago

Did Git pull and did add the verify=False to line 99 of the Google_Search.py. Added the print(Google_Item_URL) underneath line 126. Now this seems to work, at least I did get 2 new results when running the tasks. Logs below. There is Google_Search - Failed to match regular expression warnings though (in addition to those obvios from not checking the certificate)..

INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:01] "GET /tasks HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:01] "GET /static/css/template.css HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:01] "GET /static/js/main.js HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:01] "GET /static/js/jquery-3.4.1.min.js HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:01] "GET /static/js/datatables.min.js HTTP/1.1" 200 - INFO:root:2020-10-29 16:28:02 General Library - Main file created. /usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host 'servernamewasheret'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings warnings.warn( INFO:root:2020-10-29 16:28:02 General Library - File: /home/mehmeies/Scrummage/lib/static/protected/output/google/2020/10/29/Google-Query-Usernamewashere-Kaijanen-1-.net.html created. INFO:root:2020-10-29 16:28:02 General Library - Adding item to Scrummage database and other configured outputs. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading CSV configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading DOCX configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading Scrummage's Main Database configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading Elasticsearch configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading DefectDojo configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading Scumblr configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading RTIR configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading JIRA configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading email configuration data. INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:02] "GET /static/images/search_grey.png HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:02] "GET /static/images/sort_asc.png HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:02] "GET /static/images/sort_both.png HTTP/1.1" 200 - INFO:root:2020-10-29 16:28:03 Connectors Library - Email Sent. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading Slack configuration data. /usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host 'servernamewasheret'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings warnings.warn( INFO:root:2020-10-29 16:28:03 General Library - File already exists, skipping creation. INFO:root:2020-10-29 16:28:03 General Library - Adding item to Scrummage database and other configured outputs. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading CSV configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading DOCX configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading Scrummage's Main Database configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading Elasticsearch configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading DefectDojo configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading Scumblr configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading RTIR configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading JIRA configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading email configuration data. INFO:root:2020-10-29 16:28:04 Connectors Library - Email Sent. INFO:root:2020-10-29 16:28:04 Connectors Library - Loading Slack configuration data. WARNING:root:2020-10-29 16:28:04 - INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:01] "GET /tasks HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:01] "GET /static/css/template.css HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:01] "GET /static/js/main.js HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:01] "GET /static/js/jquery-3.4.1.min.js HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:01] "GET /static/js/datatables.min.js HTTP/1.1" 200 - INFO:root:2020-10-29 16:28:02 General Library - Main file created. /usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host 'servernamewasheret'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings warnings.warn( INFO:root:2020-10-29 16:28:02 General Library - File: /home/mehmeies/Scrummage/lib/static/protected/output/google/2020/10/29/Google-Query-Usernamewashere-Kaijanen-1-.net.html created. INFO:root:2020-10-29 16:28:02 General Library - Adding item to Scrummage database and other configured outputs. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading CSV configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading DOCX configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading Scrummage's Main Database configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading Elasticsearch configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading DefectDojo configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading Scumblr configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading RTIR configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading JIRA configuration data. INFO:root:2020-10-29 16:28:02 Connectors Library - Loading email configuration data. INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:02] "GET /static/images/search_grey.png HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:02] "GET /static/images/sort_asc.png HTTP/1.1" 200 - INFO:werkzeug:192.168.1.105 - - [29/Oct/2020 16:28:02] "GET /static/images/sort_both.png HTTP/1.1" 200 - INFO:root:2020-10-29 16:28:03 Connectors Library - Email Sent. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading Slack configuration data. /usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host 'servernamewasheret'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings warnings.warn( INFO:root:2020-10-29 16:28:03 General Library - File already exists, skipping creation. INFO:root:2020-10-29 16:28:03 General Library - Adding item to Scrummage database and other configured outputs. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading CSV configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading DOCX configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading Scrummage's Main Database configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading Elasticsearch configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading DefectDojo configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading Scumblr configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading RTIR configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading JIRA configuration data. INFO:root:2020-10-29 16:28:03 Connectors Library - Loading email configuration data. INFO:root:2020-10-29 16:28:04 Connectors Library - Email Sent. INFO:root:2020-10-29 16:28:04 Connectors Library - Loading Slack configuration data. WARNING:root:2020-10-29 16:28:04 - Google_Search - Failed to match regular expression. https://digital-forensics.sans.org/community/lethal-forensicator/coin-holders/. https://digital-forensics.sans.org/community/lethal-forensicator/coin-holders/

matamorphosis commented 3 years ago

Thanks for that, I have pushed an update to GitHub which should fix the regex issue, my regex basically wasn't allowing hyphens in URLs. If you have any more URLs that fail to match ofc feel free to report it and I'll be sure to add it.

As for the SSL verification issue, I'm planning on an upgrade across all plugins to automatically handle this. I'd only ever experienced it in the Domain_Fuzzer.py and Phishstats_Search.py plugins due to the high-risk nature of the results. But I was already in the mindset of moving the request handling to the General.py library, which I can extend to handle this automatically. Give me some time to sort this out and test it works. At which point I'll come back to this thread and make a new comment.

Thanks again for supporting the procurement of this plugin.

matamorphosis commented 3 years ago

I have been able to centralise the managing of requests which should now handle the removal of SSL verification for you. Let me know if the latest update resolves this issue.

matamorphosis commented 3 years ago

As there has been no further update provided I will assume this issue has been fixed and will close this off.