search

thinkst / opencanary

Modular and decentralised honeypot
http://opencanary.org
BSD 3-Clause "New" or "Revised" License
2.21k stars 353 forks source link

OpenCanary failing to start #342

Closed SecuriLee closed 4 months ago

SecuriLee commented 4 months ago

4 hours ago my 3 OCs - running for quite some time - started to choke. They all run on Ubuntu and have pro status on. At 6am CET this morning, they stopped working properly and due to a script that checks if telnet is enabled, started rebooting themselves over and over again.

I assume some update pushed through the Ubuntu pro channel was delivered and now stops the birds from working.

I am of course happy to help find the problem here and then solve it and share it on. I tried pip install opencanary -U and updated all packages on the machines, rebooted and they are still not starting.

opencanary@digger:~$ env/bin/opencanaryd --start We hope you enjoy using OpenCanary. For more open source Canary goodness, head over to canarytokens.org. [-] Failed to open opencanary.conf for reading ([Errno 2] No such file or directory: 'opencanary.conf') [-] Failed to open /home/opencanary/.opencanary.conf for reading ([Errno 2] No such file or directory: '/home/opencanary/.opencanary.conf') [-] Using config file: /etc/opencanaryd/opencanary.conf /home/opencanary/env/lib/python3.10/site-packages/twisted/conch/ssh/transport.py:97: CryptographyDeprecationWarning: Blowfish has been deprecated b"blowfish-cbc": (algorithms.Blowfish, 16, modes.CBC), /home/opencanary/env/lib/python3.10/site-packages/twisted/conch/ssh/transport.py:101: CryptographyDeprecationWarning: CAST5 has been deprecated b"cast128-cbc": (algorithms.CAST5, 16, modes.CBC), /home/opencanary/env/lib/python3.10/site-packages/twisted/conch/ssh/transport.py:106: CryptographyDeprecationWarning: Blowfish has been deprecated b"blowfish-ctr": (algorithms.Blowfish, 16, modes.CTR), /home/opencanary/env/lib/python3.10/site-packages/twisted/conch/ssh/transport.py:107: CryptographyDeprecationWarning: CAST5 has been deprecated b"cast128-ctr": (algorithms.CAST5, 16, modes.CTR), {"dst_host": "", "dst_port": -1, "local_time": "2024-02-26 09:22:08.964376", "local_time_adjusted": "2024-02-26 10:22:08.964398", "logdata": {"msg": {"logdata": "Added service from class CanaryFTP in opencanary.modules.ftp to fake"}}, "logtype": 1001, "node_id": "Switzerland", "src_host": "", "src_port": -1, "utc_time": "2024-02-26 09:22:08.964394"} ^CTraceback (most recent call last): File "/home/opencanary/env/bin/twistd", line 8, in sys.exit(run()) File "/home/opencanary/env/lib/python3.10/site-packages/twisted/scripts/twistd.py", line 35, in run app.run(runApp, ServerOptions) File "/home/opencanary/env/lib/python3.10/site-packages/twisted/application/app.py", line 674, in run runApp(config) File "/home/opencanary/env/lib/python3.10/site-packages/twisted/scripts/twistd.py", line 29, in runApp runner.run() File "/home/opencanary/env/lib/python3.10/site-packages/twisted/application/app.py", line 370, in run self.application = self.createOrGetApplication() File "/home/opencanary/env/lib/python3.10/site-packages/twisted/application/app.py", line 437, in createOrGetApplication application = getApplication(self.config, passphrase) File "/home/opencanary/env/lib/python3.10/site-packages/twisted/application/app.py", line 446, in getApplication application = service.loadApplication(filename, style, passphrase) File "/home/opencanary/env/lib/python3.10/site-packages/twisted/application/service.py", line 404, in loadApplication application = sob.loadValueFromFile(filename, "application") File "/home/opencanary/env/lib/python3.10/site-packages/twisted/persisted/sob.py", line 177, in loadValueFromFile eval(codeObj, d, d) File "/home/opencanary/env/bin/opencanary.tac", line 171, in start_mod(application, klass) File "/home/opencanary/env/bin/opencanary.tac", line 127, in start_mod logMsg({"logdata": msg}) File "/home/opencanary/env/bin/opencanary.tac", line 146, in logMsg logger.log(data, retry=False) File "/home/opencanary/env/lib/python3.10/site-packages/opencanary/logger.py", line 182, in log self.logger.warn(json.dumps(logdata, sort_keys=True)) File "/usr/lib/python3.10/logging/init.py", line 1494, in warn self.warning(msg, *args, kwargs) File "/usr/lib/python3.10/logging/init.py", line 1489, in warning self._log(WARNING, msg, args, kwargs) File "/usr/lib/python3.10/logging/init.py", line 1624, in _log self.handle(record) File "/usr/lib/python3.10/logging/init.py", line 1634, in handle self.callHandlers(record) File "/usr/lib/python3.10/logging/init.py", line 1696, in callHandlers hdlr.handle(record) File "/usr/lib/python3.10/logging/init.py", line 968, in handle self.emit(record) File "/home/opencanary/env/lib/python3.10/site-packages/opencanary/logger.py", line 366, in emit response = requests.request( File "/home/opencanary/env/lib/python3.10/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, kwargs) File "/home/opencanary/env/lib/python3.10/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/home/opencanary/env/lib/python3.10/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/home/opencanary/env/lib/python3.10/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/home/opencanary/env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 597, in urlopen httplib_response = self._make_request(conn, method, url, File "/home/opencanary/env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 354, in _make_request conn.request(method, url, httplib_request_kw) File "/usr/lib/python3.10/http/client.py", line 1283, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 976, in send self.connect() File "/home/opencanary/env/lib/python3.10/site-packages/urllib3/connection.py", line 181, in connect conn = self._new_conn() File "/home/opencanary/env/lib/python3.10/site-packages/urllib3/connection.py", line 158, in _new_conn conn = connection.create_connection( File "/home/opencanary/env/lib/python3.10/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) KeyboardInterrupt

SecuriLee commented 4 months ago

OK, I nuked an OC instance in GCP today, removed Live Patch but updated Ubuntu 22.04 LTS to latest and did a bare install. The OC will not start.

It started with a --copyconfig conf file so it seems the issue is my config; I actually tracked this down to the webhooks that were being sent to my Splunk over Tailscale - somehow not working but causing a massive pause in the service coming up.

Question: if a logging destination is not available, can the OC still come up? I am happy too, to close this if it's not possible.

thinkst-pieter commented 4 months ago

Hi @SecuriLee sorry for the delayed response and thanks for the update on the issue.

It started with a --copyconfig conf file so it seems the issue is my config; I actually tracked this down to the webhooks that were being sent to my Splunk over Tailscale - somehow not working but causing a massive pause in the service coming up.

Glad you were able to sort your config and get OC running again.

Question: if a logging destination is not available, can the OC still come up?

That is a great question. Ideally you would want a clear indication that a log destination is not available and not have it pause the service from coming up. Lets flag it for some discussion.