search

duosecurity / duo_log_sync

MIT License
37 stars 27 forks source link

adminaction consumer: failed to write some logs #17

Open jsandova opened 3 years ago

jsandova commented 3 years ago

I am getting the following error about adminaction unable to write to logs. Any ideas?

2020-12-22 23:19:50 INFO Starting DuoLogSync 2020-12-22 23:19:50 INFO DuoLogSync: Opening connection to ls01-dev-qa.aofk.net:2514 2020-12-22 23:19:50 INFO duo_client Admin initialized for ikey: ***, host: api-**.duosecurity.com 2020-12-22 23:19:50 ERROR Could not read checkpoint file for adminaction logs, consuming logs from {log_offset} timestamp 2020-12-22 23:19:50 ERROR Could not read checkpoint file for auth logs, consuming logs from {log_offset} timestamp 2020-12-22 23:19:50 INFO adminaction producer: fetching next logs after 120 seconds 2020-12-22 23:19:50 INFO adminaction consumer: waiting for logs 2020-12-22 23:19:50 INFO auth producer: fetching next logs after 120 seconds 2020-12-22 23:19:50 INFO auth consumer: waiting for logs 2020-12-22 23:21:50 INFO adminaction producer: fetching logs 2020-12-22 23:21:50 INFO auth producer: fetching logs Traceback (most recent call last): 2020-12-22 23:21:50 INFO adminaction producer: adding 57 logs to the queue 2020-12-22 23:21:50 INFO adminaction producer: added 57 logs to the queue 2020-12-22 23:21:50 INFO adminaction producer: fetching next logs after 120 seconds 2020-12-22 23:21:50 INFO adminaction consumer: received 57 logs from producer 2020-12-22 23:21:50 INFO adminaction consumer: writing logs 2020-12-22 23:21:50 WARNING adminaction consumer: failed to write some logs File "/usr/local/lib/python3.6/dist-packages/duologsync-2.0.0-py3.6.egg/duologsync/consumer/consumer.py", line 66, in consume File "/usr/local/lib/python3.6/dist-packages/duologsync-2.0.0-py3.6.egg/duologsync/writer.py", line 97, in write File "/usr/lib/python3.6/asyncio/streams.py", line 329, in drain raise exc File "/usr/lib/python3.6/asyncio/selector_events.py", line 714, in _read_ready data = self._sock.recv(self.max_size) ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/bin/duologsync", line 11, in load_entry_point('duologsync==2.0.0', 'console_scripts', 'duologsync')() File "/usr/local/lib/python3.6/dist-packages/duologsync-2.0.0-py3.6.egg/duologsync/app.py", line 78, in main File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete return future.result() File "/usr/local/lib/python3.6/dist-packages/duologsync-2.0.0-py3.6.egg/duologsync/consumer/consumer.py", line 88, in consume File "/usr/local/lib/python3.6/dist-packages/duologsync-2.0.0-py3.6.egg/duologsync/producer/producer.py", line 205, in get_log_offset TypeError: 'NoneType' object is not subscriptable

jsandova commented 3 years ago

Here is my config.yml file.

version: '1.0.0' dls_settings: log_format: 'JSON' api: offset: 1 checkpointing: enabled: True directory: '/var/log/duo-logs' servers:

jsandova commented 3 years ago

I was able to get it working by switching to UDP and using fluentd to forward the logs to our datadog logging console.

rka commented 3 years ago

I had a similar issue and also solved it by using UDP. Really annoying because we would like to use TCP :/