INFO:pastehunter.py:Unable to retrieve paste, no uri found.

[ahezza]

I've noticed after a couple of days of running Pastehunter as a service in the background on my Ubuntu Server, the process seems to time out then never come back. The only way to get it running again is to restart the service. I've checked my syslog and noticed the following right before it stops running:

Dec  8 08:22:32 hostname python3[18445]: INFO:pastehunter.py:Sleeping for 300 Seconds
Dec  8 08:22:51 hostname python3[18445]: INFO:pastehunter.py:Running Post Module postprocess.post_email on 11024631823
Dec  8 08:23:01 hostname python3[18445]: INFO:pastehunter.py:Running Post Module postprocess.post_email on 11024631751
Dec  8 08:23:02 hostname python3[18445]: INFO:pastehunter.py:Running Post Module postprocess.post_email on 11024631751
Dec  8 08:23:12 hostname python3[18445]: INFO:pastehunter.py:Blacklisted github.com paste 11024631694
Dec  8 08:23:14 hostname python3[18445]: message repeated 3 times: [ INFO:pastehunter.py:Blacklisted github.com paste 11024631694]
Dec  8 08:23:19 hostname python3[18445]: INFO:pastehunter.py:Blacklisted github.com paste 11024631674
Dec  8 08:23:24 hostname python3[18445]: message repeated 18 times: [ INFO:pastehunter.py:Blacklisted github.com paste 11024631674]
Dec  8 08:24:23 hostname python3[18445]: INFO:pastehunter.py:Blacklisted github.com paste 11024631631
Dec  8 08:24:30 hostname python3[18445]: message repeated 2 times: [ INFO:pastehunter.py:Blacklisted github.com paste 11024631631]
Dec  8 08:24:47 hostname python3[18445]: INFO:pastehunter.py:Blacklisted github.com paste 11024631566
Dec  8 08:25:05 hostname python3[18445]: message repeated 17 times: [ INFO:pastehunter.py:Blacklisted github.com paste 11024631566]
Dec  8 08:25:13 hostname python3[18445]: ERROR:pastehunter.py:Unable to scan raw paste : s259FR8dIF - external values must be of type integer, float, boolean or string
Dec  8 08:25:13 hostname python3[18445]: ERROR:pastehunter.py:Unable to scan raw paste : s2Ih7E6KrE - external values must be of type integer, float, boolean or string
Dec  8 08:25:17 hostname python3[18445]: INFO:pastehunter.py:Running Post Module postprocess.post_email on vQZAQe8s
Dec  8 08:25:18 hostname python3[18445]: INFO:pastehunter.py:Blacklisted pastebin.com paste 6PZ22H2Q
Dec  8 08:25:20 hostname python3[18445]: INFO:pastehunter.py:Running Post Module postprocess.post_email on TU90Wm1P
Dec  8 08:25:21 hostname python3[18445]: INFO:pastehunter.py:Blacklisted pastebin.com paste VFCBDwty
Dec  8 08:25:23 hostname python3[18445]: INFO:pastehunter.py:Blacklisted pastebin.com paste 3G5N9EfH
Dec  8 08:25:27 hostname python3[18445]: INFO:pastehunter.py:Running Post Module postprocess.post_email on 11024660038
Dec  8 08:25:51 hostname python3[18445]: Process timeout: Timeout
Dec  8 08:26:03 hostname python3[18445]: INFO:pastehunter.py:Unable to retrieve paste, no uri found.
Dec  8 08:26:03 hostname python3[18445]: message repeated 17 times: [ INFO:pastehunter.py:Unable to retrieve paste, no uri found.]
Dec  8 08:26:14 hostname python3[18445]: INFO:pastehunter.py:Blacklisted github.com paste 11024659930
Dec  8 08:26:14 hostname python3[18445]: INFO:pastehunter.py:Blacklisted github.com paste 11024659930
Dec  8 08:26:19 hostname python3[18445]: INFO:pastehunter.py:Blacklisted github.com paste 11024659905
Dec  8 08:26:37 hostname python3[18445]: message repeated 13 times: [ INFO:pastehunter.py:Blacklisted github.com paste 11024659905]
Dec  8 08:26:49 hostname python3[18445]: INFO:pastehunter.py:Blacklisted github.com paste 11024659883
Dec  8 08:26:50 hostname python3[18445]: INFO:pastehunter.py:Running Post Module postprocess.post_email on 11024659868
Dec  8 08:27:31 hostname python3[18445]: INFO:pastehunter.py:Blacklisted github.com paste 11024659822
Dec  8 08:27:32 hostname python3[18445]: WARNING:pastehunter.py:Restarting Dead Process
Dec  8 08:27:36 hostname python3[18445]: Process timeout: Timeout

I'm guessing it's the repeated message "Unable to retrieve paste, no uri found." that has stopped it functioning correctly, but no sure exactly was the issue is.. Let me know if you need any more information.

[Plazmaz]

So it's a little bit difficult to tell for sure. Can you give me some information on what engines you've got enabled? I see github and pastebin. A few pieces here:

1. It looks like the process is timing out, causing the issue.
2. My best guess for Unable to retrieve paste, no uri found. is that it's due to deleted/changed github commits or users.
3. If you have Slexy enabled, it is quite possible that timeouts are occurring due to high request volume from pastehunter. We're working with them to improve that. For now, you may want to disable it.

If you could send over a copy of your config (with keys/email/etc removed), I can take a look and see if anything sticks out.

[ahezza]

Here's a copy of my config:

{
  "inputs": {
    "pastebin":{
      "enabled": true,
      "module": "inputs.pastebin",
      "api_scrape": "https://scrape.pastebin.com/api_scraping.php",
      "api_raw": "https://scrape.pastebin.com/api_scrape_item.php?i=",
      "paste_limit": 200,
      "store_all": false
    },
    "dumpz": {
      "enabled": false,
      "comment": "This api endpoint has been removed.",
      "module": "inputs.dumpz",
      "api_scrape": "https://dumpz.org/api/recent",
      "api_raw": "https://dumpz.org/api/dump",
      "paste_limit": 200,
      "store_all": false
    },
    "gists": {
      "enabled": true,
      "module": "inputs.gists",
      "api_token": "<omitted>",
      "api_limit": 100,
      "store_all": false,
      "user_blacklist": [],
      "file_blacklist": ["grahamcofborg-eval-package-list", "Changed Paths"]
    },
    "github": {
      "enabled": true,
      "module": "inputs.github",
      "api_token": "<omitted>",
      "api_limit": 100,
      "store_all": false,
      "user_blacklist": [],
      "file_blacklist": ["node_modules/*", "__pycache__/*", "*/grahamcofborg-eval-package-list", "*/yarn.lock", "*.3ds", "*.3g2", "*.3gp", "*.7z", "*.DS_Store", "*.a", "*.aac", "*.adp", "*.ai", "*.aif", "*.aiff", "*.alz", "*.ape", "*.apk", "*.ar", "*.arj", "*.asf", "*.au", "*.avi", "*.bak", "*.bh", "*.bin", "*.bk", "*.bmp", "*.btif", "*.bz2", "*.bzip2", "*.cab", "*.caf", "*.cgm", "*.class", "*.cmx", "*.cpio", "*.cr2", "*.csv", "*.cur", "*.dat", "*.deb", "*.dex", "*.djvu", "*.dll", "*.dmg", "*.dng", "*.doc", "*.docm", "*.docx", "*.dot", "*.dotm", "*.dra", "*.dsk", "*.dts", "*.dtshd", "*.dvb", "*.dwg", "*.dxf", "*.ecelp4800", "*.ecelp7470", "*.ecelp9600", "*.egg", "*.eol", "*.eot", "*.epub", "*.exe", "*.f4v", "*.fbs", "*.fh", "*.fla", "*.flac", "*.fli", "*.flv", "*.fpx", "*.fst", "*.fvt", "*.g3", "*.gif", "*.graffle", "*.gz", "*.gzip", "*.h261", "*.h263", "*.h264", "*.ico", "*.ief", "*.img", "*.ipa", "*.iso", "*.jar", "*.jpeg", "*.jpg", "*.jpgv", "*.jpm", "*.jxr", "*.key", "*.ktx", "*.lha", "*.lvp", "*.lz", "*.lzh", "*.lzma", "*.lzo", "*.m3u", "*.m4a", "*.m4v", "*.mar", "*.mdi", "*.mht", "*.mid", "*.midi", "*.mj2", "*.mka", "*.mkv", "*.mmr", "*.mng", "*.mobi", "*.mov", "*.movie", "*.mp3", "*.mp4", "*.mp4a", "*.mpeg", "*.mpg", "*.mpga", "*.mxu", "*.nef", "*.npx", "*.numbers", "*.o", "*.oga", "*.ogg", "*.ogv", "*.otf", "*.pages", "*.pbm", "*.pcx", "*.pdf", "*.pea", "*.pgm", "*.pic", "*.png", "*.pnm", "*.pot", "*.potm", "*.potx", "*.ppa", "*.ppam", "*.ppm", "*.pps", "*.ppsm", "*.ppsx", "*.ppt", "*.pptm", "*.pptx", "*.psd", "*.pya", "*.pyc", "*.pyo", "*.pyv", "*.qt", "*.rar", "*.ras", "*.raw", "*.rgb", "*.rip", "*.rlc", "*.rmf", "*.rmvb", "*.rtf", "*.rz", "*.s3m", "*.s7z", "*.scpt", "*.sgi", "*.shar", "*.sil", "*.sketch", "*.slk", "*.smv", "*.so", "*.sub", "*.swf", "*.tar", "*.tbz", "*.tbz2", "*.tga", "*.tgz", "*.thmx", "*.tif", "*.tiff", "*.tlz", "*.ttc", "*.ttf", "*.txz", "*.udf", "*.uvh", "*.uvi", "*.uvm", "*.uvp", "*.uvs", "*.uvu", "*.viv", "*.vob", "*.war", "*.wav", "*.wax", "*.wbmp", "*.wdp", "*.weba", "*.webm", "*.webp", "*.whl", "*.wim", "*.wm", "*.wma", "*.wmv", "*.wmx", "*.woff", "*.woff2", "*.wvx", "*.xbm", "*.xif", "*.xla", "*.xlam", "*.xls", "*.xlsb", "*.xlsm", "*.xlsx", "*.xlt", "*.xltm", "*.xltx", "*.xm", "*.xmind", "*.xpi", "*.xpm", "*.xwd", "*.xz", "*.z", "*.zip", "*.zipx"]
    },
    "slexy":{
      "enabled": true,
      "module": "inputs.slexy",
      "store_all": false,
      "api_scrape": "http://slexy.org/recent",
      "api_raw": "http://slexy.org/raw",
      "api_view": "http://slexy.org/view"
    },
    "stackexchange":{
      "enabled": true,
      "module": "inputs.stackexchange",
      "site_list": ["stackoverflow","serverfault", "superuser", "webapps", "webmasters", "dba"],
      "api_key": "<omitted>",
      "store_filter": "!)r_ttsG0v3bE1vo3*8Ki",
      "pagesize": 100,
      "store_all": true,
      "api_scrape": "https://api.stackexchange.com/2.2/questions"
    }
  },
  "outputs": {
    "elastic_output": {
      "enabled": false,
      "module": "outputs.elastic_output",
      "classname": "ElasticOutput",
      "elastic_index": "paste-test",
      "elastic_host": "172.16.10.10",
      "elastic_port": 9200,
      "elastic_user": "elastic",
      "elastic_pass": "changeme",
      "elastic_ssl": false,
      "weekly_index": true
    },
    "splunk_output": {
      "enabled": true,
      "module": "outputs.splunk_output",
      "classname": "SplunkOutput",
      "splunk_host": "<omitted>",
      "splunk_port": <omitted>,
      "splunk_user": "<omitted>",
      "splunk_pass": "<omitted>",
      "splunk_index": "<omitted>",
      "splunk_sourcetype": "<omitted>",
      "store_raw": true
    },
    "json_output": {
      "enabled": false,
      "module": "outputs.json_output",
      "classname": "JsonOutput",
      "output_path": "logs/json/",
      "store_raw": true,
      "encode_raw": true
    },
    "csv_output": {
      "enabled": false,
      "module": "outputs.csv_output",
      "classname": "CSVOutput",
      "output_path": "logs/csv/"
    },
    "syslog_output": {
      "enabled": false,
      "module": "outputs.syslog_output",
      "classname": "SyslogOutput",
      "host": "192.168.1.1",
      "port": 514
    },
    "smtp_output": {
      "enabled": true,
      "module": "outputs.smtp_output",
      "classname": "SMTPOutput",
      "smtp_host": "<omitted>",
      "smtp_port": <omitted>,
      "smtp_security": "<omitted>",
      "smtp_user": "<omitted>",
      "smtp_pass": "<omitted>",
      "recipients" : {
        "recipient_1": {
          "address": "<omitted>,
          "rule_list": ["<omitted>", "<omitted>", "<omitted>"],
          "mandatory_rule_list": []
        },
        "recipient_2": {
          "address": "<omitted>",
          "rule_list": ["<omitted>", "<omitted>", "<omitted>"],
          "mandatory_rule_list": []
        }
      }
    },
    "slack_output": {
      "enabled": false,
      "module": "outputs.slack_output",
      "classname": "SlackOutput",
      "webhook_url": "",
      "rule_list": ["custom_keywords"]
    },
    "twilio_output": {
      "enabled": false,
      "module": "outputs.twilio_output",
      "classname": "TwilioOutput",
      "account_sid": "",
      "auth_token": "",
      "twilio_sender": "",
      "recipient_list": [],
      "rule_list": ["custom_keywords"]
    }
  },
  "yara": {
    "rule_path": "YaraRules",
    "blacklist": true,
    "test_rules": false
  },
  "log": {
    "log_to_file": false,
    "log_file": "pastehunter",
    "logging_level": 20,
    "log_path": "logs",
    "format": "%(asctime)s [%(threadName)-12.12s] %(levelname)s:%(message)s"
  },
  "general": {
    "run_frequency": 300,
    "process_timeout": 5
  },
  "sandboxes": {
    "cuckoo": {
      "enabled": false,
      "module": "sandboxes.cuckoo",
      "api_host": "127.0.0.1",
      "api_port": 8080
    },
    "viper": {
      "enabled": false,
      "module": "sandboxes.viper",
      "api_host": "127.0.0.1",
      "api_port": 8080
    }
  },
  "post_process": {
    "post_email": {
      "enabled": true,
      "module": "postprocess.post_email",
      "rule_list": ["email_list"]
    },
    "post_b64": {
      "enabled": true,
      "module": "postprocess.post_b64",
      "rule_list": ["b64_exe", "b64_rar", "b64_zip", "b64_gzip"]
    },
    "post_entropy": {
      "enabled": false,
      "module": "postprocess.post_entropy",
      "rule_list": ["ALL"]
    },
    "post_compress": {
      "enabled": false,
      "module": "postprocess.post_compress",
      "rule_list": ["ALL"]
    }
  }
}

Many thanks,

[ahezza]

I have also disabled slexy based on your advice earlier, that line now reads as "false" in the config above

[Plazmaz]

@ahezza did disabling slexy resolve your issue here?

[ahezza]

No luck sadly. It went down again yesterday and required a manual restart to get it going again..

On 10 Dec 2019, at 21:45, Dylan Katz notifications@github.com wrote:

 @ahezza did disabling slexy resolve your issue here?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[Plazmaz]

In order to gather more information on this, would you mind enabling debug logs? You can do that by changing this section of your config:

  "log": {
    "log_to_file": false,
    "log_file": "pastehunter",
    "logging_level": 20,
    "log_path": "logs",
    "format": "%(asctime)s [%(threadName)-12.12s] %(levelname)s:%(message)s"
  },

to this:

  "log": {
    "log_to_file": false,
    "log_file": "pastehunter",
    "logging_level": 10,
    "log_path": "logs",
    "format": "%(asctime)s [%(threadName)-12.12s] %(levelname)s:%(message)s"
  },

Then you should be able to see the paste causing that error message.

Also, just double checking since you've got stackexchange enabled, are you running from the latest source? There was a bug fixed in 9cfa32df08dc7e711f8765c71d256dcd7efe74e2 that had similar symptoms (see #72)

[ahezza]

Thanks Dylan,

I have changed the logging level like you've suggested and restarted the process. I will let you know when it crashes again and paste the output of the logs. Hopefully, that should tell us the root cause.

I'm running the latest version of Pastehunter as of 22nd November 2019 as that is the date I reinstalled my instance.

On Fri, Dec 13, 2019 at 12:45 AM Dylan Katz notifications@github.com wrote:

In order to gather more information on this, would you mind enabling debug logs? You can do that by changing this section of your config:

"log": { "log_to_file": false, "log_file": "pastehunter", "logging_level": 20, "log_path": "logs", "format": "%(asctime)s [%(threadName)-12.12s] %(levelname)s:%(message)s" },

to this:

"log": { "log_to_file": false, "log_file": "pastehunter", "logging_level": 10, "log_path": "logs", "format": "%(asctime)s [%(threadName)-12.12s] %(levelname)s:%(message)s" },

Then you should be able to see the paste causing that error message.

Also, just double checking since you've got stackexchange enabled, are you running from the latest source? There was a bug fixed in 9cfa32d https://github.com/kevthehermit/PasteHunter/commit/9cfa32df08dc7e711f8765c71d256dcd7efe74e2 that had similar symptoms (see #72 https://github.com/kevthehermit/PasteHunter/issues/72)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kevthehermit/PasteHunter/issues/97?email_source=notifications&email_token=AGH5QME3IFNUVFAESN7IPC3QYLLK5A5CNFSM4JX2U7TKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGYRAQY#issuecomment-565252163, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGH5QMFZ2QK6I2Y5OUVSES3QYLLK5ANCNFSM4JX2U7TA .

[ahezza]

My pastehunter instance stopped running yesterday. Here's the output from the service status and the last 50 lines of the log file:

user@[redacted]:~$ sudo systemctl status pastehunter.service

● pastehunter.service - PasteHunter
   Loaded: loaded (/etc/systemd/system/pastehunter.service; disabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-12-13 10:54:18 CET; 2 days ago
 Main PID: 9624 (python3)
    Tasks: 2 (limit: 4588)
   CGroup: /system.slice/pastehunter.service
           └─9624 /usr/bin/python3 /opt/pastehunter/pastehunter.py

Dec 15 17:27:32 [redacted] python3[9624]: DEBUG:pastehunter.py:Processing Finished for 11085645945 in 0.5131235122680664 seconds
Dec 15 17:27:32 [redacted] python3[9624]: DEBUG:pastehunter.py:Found New github.com paste 11085645945
Dec 15 17:27:32 [redacted] python3[9624]: DEBUG:pastehunter.py:Processing Finished for 11085645945 in 0.5375335216522217 seconds
Dec 15 17:27:32 [redacted] python3[9624]: DEBUG:pastehunter.py:Found New github.com paste 11085645945
Dec 15 17:27:37 [redacted] python3[9624]: Process timeout: Timeout
Dec 15 17:27:37 [redacted] python3[9624]: DEBUG:pastehunter.py:Processing Finished for 11085645945 in 4.76540207862854 seconds
Dec 15 17:27:37 [redacted] python3[9624]: DEBUG:pastehunter.py:Found New github.com paste 11085645945
Dec 15 17:27:39 [redacted] python3[9624]: DEBUG:pastehunter.py:Processing Finished for 11085645945 in 2.4369313716888428 seconds
Dec 15 17:27:39 [redacted] python3[9624]: DEBUG:pastehunter.py:Found New github.com paste 11085645945
Dec 15 17:27:44 [redacted] python3[9624]: Process timeout: Timeout

user@[redacted]:~$ tail -n50 /opt/pastehunter/logs/pastehunter.log       
2019-12-15 17:27:26,516 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.5198569297790527 seconds
2019-12-15 17:27:26,517 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:26,719 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.4574763774871826 seconds
2019-12-15 17:27:26,720 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:27,151 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.4304676055908203 seconds
2019-12-15 17:27:27,152 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:27,301 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.7843246459960938 seconds
2019-12-15 17:27:27,303 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:27,603 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.45162081718444824 seconds
2019-12-15 17:27:27,604 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:27,782 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.47959470748901367 seconds
2019-12-15 17:27:27,784 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:28,076 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.4714014530181885 seconds
2019-12-15 17:27:28,077 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:28,252 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.4677097797393799 seconds
2019-12-15 17:27:28,253 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:28,796 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.719003438949585 seconds
2019-12-15 17:27:28,798 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:29,023 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.7700729370117188 seconds
2019-12-15 17:27:29,028 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:29,253 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.45525097846984863 seconds
2019-12-15 17:27:29,254 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:29,559 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.5310871601104736 seconds
2019-12-15 17:27:29,563 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:29,772 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.5174822807312012 seconds
2019-12-15 17:27:29,773 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:30,121 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.5580263137817383 seconds
2019-12-15 17:27:30,122 [MainThread  ] DEBUG:Found New github.com paste 11085645990
2019-12-15 17:27:30,289 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.5151760578155518 seconds
2019-12-15 17:27:30,290 [MainThread  ] DEBUG:Found New github.com paste 11085645960
2019-12-15 17:27:30,579 [MainThread  ] DEBUG:Processing Finished for 11085645990 in 0.45673441886901855 seconds
2019-12-15 17:27:30,584 [MainThread  ] DEBUG:Found New github.com paste 11085645960
2019-12-15 17:27:31,023 [MainThread  ] DEBUG:Processing Finished for 11085645960 in 0.7336015701293945 seconds
2019-12-15 17:27:31,024 [MainThread  ] DEBUG:Found New github.com paste 11085645960
2019-12-15 17:27:31,166 [MainThread  ] DEBUG:Processing Finished for 11085645960 in 0.581573486328125 seconds
2019-12-15 17:27:31,167 [MainThread  ] DEBUG:Found New github.com paste 11085645946
2019-12-15 17:27:31,648 [MainThread  ] DEBUG:Processing Finished for 11085645946 in 0.4808487892150879 seconds
2019-12-15 17:27:31,649 [MainThread  ] DEBUG:Found New github.com paste 11085645946
2019-12-15 17:27:31,679 [MainThread  ] DEBUG:Processing Finished for 11085645960 in 0.6545729637145996 seconds
2019-12-15 17:27:31,680 [MainThread  ] DEBUG:Found New github.com paste 11085645945
2019-12-15 17:27:32,093 [MainThread  ] DEBUG:Processing Finished for 11085645946 in 0.4440443515777588 seconds
2019-12-15 17:27:32,094 [MainThread  ] DEBUG:Found New github.com paste 11085645945
2019-12-15 17:27:32,193 [MainThread  ] DEBUG:Processing Finished for 11085645945 in 0.5131235122680664 seconds
2019-12-15 17:27:32,194 [MainThread  ] DEBUG:Found New github.com paste 11085645945
2019-12-15 17:27:32,632 [MainThread  ] DEBUG:Processing Finished for 11085645945 in 0.5375335216522217 seconds
2019-12-15 17:27:32,633 [MainThread  ] DEBUG:Found New github.com paste 11085645945
2019-12-15 17:27:37,399 [MainThread  ] DEBUG:Processing Finished for 11085645945 in 4.76540207862854 seconds
2019-12-15 17:27:37,400 [MainThread  ] DEBUG:Found New github.com paste 11085645945
2019-12-15 17:27:39,837 [MainThread  ] DEBUG:Processing Finished for 11085645945 in 2.4369313716888428 seconds
2019-12-15 17:27:39,839 [MainThread  ] DEBUG:Found New github.com paste 11085645945

[Plazmaz]

You might have some luck with then changes on https://github.com/kevthehermit/PasteHunter/tree/develop. I've refactored how we handle multiprocessing significantly, and it seems to be quite a bit more stable.

[ahezza]

Great, I've just grabbed the latest version and am now running it in my instance. I'll update in a few days to see if the process stops again.

On Wed, Dec 25, 2019 at 1:03 AM Dylan Katz notifications@github.com wrote:

You might have some luck with then changes on https://github.com/kevthehermit/PasteHunter/tree/develop. I've refactored how we handle multiprocessing significantly, and it seems to be quite a bit more stable.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kevthehermit/PasteHunter/issues/97?email_source=notifications&email_token=AGH5QMGDSXOCKSE5TEFO2RLQ2KWM7A5CNFSM4JX2U7TKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHTXDRI#issuecomment-568816069, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGH5QMCZ2GODU3SRTGK3PLDQ2KWM7ANCNFSM4JX2U7TA .

[ahezza]

Apologies, I should've posted sooner. I updated to the latest version, however, the process timed out once more:

● pastehunter.service - PasteHunter
   Loaded: loaded (/etc/systemd/system/pastehunter.service; disabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-12-27 15:36:48 CET; 1 weeks 2 days ago
 Main PID: 15529 (python3)
    Tasks: 2 (limit: 4588)
   CGroup: /system.slice/pastehunter.service
           └─15529 /usr/bin/python3 /opt/pastehunter/pastehunter.py

Dec 30 19:33:18 [redacted] python3[15529]: DEBUG:pastehunter.py:Found New github.com paste 11169912551
Dec 30 19:33:19 [redacted] python3[15529]: DEBUG:pastehunter.py:Processing Finished for 11169912551 in 0.4985237121582031 seconds
Dec 30 19:33:19 [redacted] python3[15529]: DEBUG:pastehunter.py:Found New github.com paste 11169912551
Dec 30 19:33:19 [redacted] python3[15529]: DEBUG:pastehunter.py:Processing Finished for 11169912551 in 0.4847538471221924 seconds
Dec 30 19:33:19 [redacted] python3[15529]: DEBUG:pastehunter.py:Found New github.com paste 11169912551
Dec 30 19:33:20 [redacted] python3[15529]: DEBUG:pastehunter.py:Processing Finished for 11169912551 in 0.47942280769348145 seconds
Dec 30 19:33:20 [redacted] python3[15529]: DEBUG:pastehunter.py:Found New github.com paste 11169912551
Dec 30 19:33:20 [redacted] python3[15529]: DEBUG:pastehunter.py:Processing Finished for 11169912551 in 0.5567107200622559 seconds
Dec 30 19:33:20 [redacted] python3[15529]: DEBUG:pastehunter.py:Found New github.com paste 11169912551
Dec 30 19:33:25 [redacted] python3[15529]: Process timeout: Timeout

Happy to provide more information, let me know what details you need.

[Plazmaz]

Yep, so I am able to reproduce, BUT, the process timeout should no longer stop the program from gathering new pastes after we changed the multiprocessing setup. (it may skip particularly large github results)

[Plazmaz]

@ahezza did this error actually cause freezing problems for you running from the latest development branch?

[ahezza]

@Plazmaz I downloaded the latest copy of Pastehunter on 27th December, and it froze a few days later. I've got it running currently with no issues. I will keep an eye on it and see if it freezes once more. Let me know if I can provide any further information/details.

[Plazmaz]

@ahezza Any problems?

[ahezza]

@Plazmaz I've had to manually restart it a couple of times. Looks like it timed out again at around 2am this morning.. I've noticed you've made some changes to Pastehunter recently, I will try downloading the latest version and see if the issue persists.

[ahezza]

Since updating to the latest version, I have had 5 days uptime. Happy to close this one off as a result, thanks

[Plazmaz]

Hooray! Happy to hear it, thanks for the update, and thanks so much for the help gathering info on this one 😄

[ahezza]

My pleasure 🙂

On 28 Jan 2020, at 18:58, Dylan Katz notifications@github.com wrote:

 Hooray! Happy to hear it, thanks for the update, and thanks so much for the help gathering info on this one 😄

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.