ParseDMARC keeps failing, unable to talk to elastic

[nkrgovic]

Running this on CentOS 8, switched SE Linux to permissive just in case, running firewalld.

parsedmarc keeps dying. I see it in docker ps output, but it dies a few seconds after restaring. I get this in logs:

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/bin/parsedmarc", line 8, in sys.exit(_main()) File "/usr/local/lib/python3.9/site-packages/parsedmarc/cli.py", line 502, in _main elastic.migrate_indexes(aggregate_indexes=[es_aggregate_index], File "/usr/local/lib/python3.9/site-packages/parsedmarc/elastic.py", line 244, in migrate_indexes if not Index(aggregate_index_name).exists(): File "/usr/local/lib/python3.9/site-packages/elasticsearch_dsl/index.py", line 414, in exists return self._get_connection(using).indices.exists(index=self._name, *kwargs) File "/usr/local/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 153, in _wrapped return func(args, params=params, headers=headers, **kwargs) File "/usr/local/lib/python3.9/site-packages/elasticsearch/client/indices.py", line 332, in exists return self.transport.perform_request( File "/usr/local/lib/python3.9/site-packages/elasticsearch/transport.py", line 413, in perform_request raise e File "/usr/local/lib/python3.9/site-packages/elasticsearch/transport.py", line 381, in perform_request status, headers_response, data = connection.perform_request( File "/usr/local/lib/python3.9/site-packages/elasticsearch/connection/http_urllib3.py", line 264, in perform_request raise ConnectionError("N/A", str(e), e) elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fa2d6ca5610>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fa2d6ca5610>: Failed to establish a new connection: [Errno 111] Connection refused)

Elastic looks OK, it's working.

Read the blog, tried moving to the older version of elastic... nothing. some advice would be appreciated.

If it matters, it's running in a VM, and the only other thing running is nginx as a proxy for http, with auth and https :).

Any advice would be appreciated.

[boolsee]

I had the same problem. At first, I think it is a some bug, but I noticed it might be a timing issue. Parsedmarc tries connect to elasticsearch before elasticsearch service fully started. After elasticsearch fully started, it gone.

[mplm17]

Hi, I've the same problem, even if elasticsearch is up, parsedmarc container cannot connect to elasticsearch:

Failed to establish a new connection: [Errno 111] Connection refused

and the container restart again and again.

[supaeasy]

Just keep it running for some time. It will work eventually.

[mplm17]

You're right! After some time the container just stop (without restarting). I was thinking it was like a timeout but I was wrong, when the container stops it means the process exited without any error and the script is working perfectly. Thanks for your reply btw ;)

[MrHappy]

How long does this usually take?

[supaeasy]

How long does this usually take?

I guess that depends on your hardware and Disk speed. On my Raspberry this takes about 1-2 Minutes. Maybe it also depends on the amount of data. That's low on my side.

[MrHappy]

How long does this usually take?

I guess that depends on your hardware and Disk speed. On my Raspberry this takes about 1-2 Minutes. Maybe it also depends on the amount of data. That's low on my side.

Thanks, looks like the issue I am having is elasticsearch failing to start...

[supaeasy]

You could try and update it, as well as docker. Related maybe: https://github.com/debricked/dmarc-visualizer/issues/19#issuecomment-890956709 IIRC I did have problems with ES, too. Maybe tryout my fork. It's been too long for me to be sure.

[MrHappy]

You could try and update it, as well as docker. Related maybe: #19 (comment) IIRC I did have problems with ES, too. Maybe tryout my fork. It's been too long for me to be sure.

I tried both your fork as this one and they both have the same issue, posted here: #20

[supaeasy]

Sorry, no idea.

[sushantrimal10]

Sorry, no idea.

Hi, I can see the grafana interface in the browser. however, can't see the statistics. I suspect it is because my elasticsearch and parsedmarc are not talking to each other. It got this error: > elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fac6e587dc0>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fac6e587dc0>: Failed to establish a new connection: [Errno 111] Connection refused) Any idea how to resolve it? Thanks

[swissroach]

I think you have to use elasticsearch:9200 as URL/Servername instead of localhost or 127.0.0.1

[bezzoh]

I'm seeing this issue as well, having just built a VM and followed these instructions. Did anyone ever get to the bottom of this problem?

[supaeasy]

Just use the fork by Luke Callaghan but keep an eye on the Customizations. It works fine without issues at all: https://github.com/LukeCallaghan/dmarc-visualizer/issues/1