Error when pop'ing messages

[fribse]

I have finally managed to get the elastic stack migrated to new hardware, and I've restarted the dmarc2logstash. There are currently 55000+ reports waiting.

But I see this message when it runs:

dmarc2filebeat        | 2023-04-25 10:47:48,405 - dmarc2logstash - INFO - Starting DMARC to Logstash service; sleepSec=600; jsonOutputFile=/dmarclogs/dmarc.json; shouldDelete=1; shouldDeleteFailures=1
dmarc2filebeat        | 2023-04-25 10:47:48,406 - dmarc2logstash - INFO - Connecting to POP3 server; server=popserver.dk; username=dmarc@popserver.dk; debugLevel=0
dmarc2filebeat        | Traceback (most recent call last):
dmarc2filebeat        |   File "/opt/dmarc2logstash/dmarc2logstash.py", line 235, in <module>
dmarc2filebeat        |     sys.exit(main())
dmarc2filebeat        |   File "/opt/dmarc2logstash/dmarc2logstash.py", line 232, in main
dmarc2filebeat        |     start(server, username, password, int(sleepSec), jsonOutputFile, float(timeout), shouldDelete, shouldDeleteFailures)
dmarc2filebeat        |   File "/opt/dmarc2logstash/dmarc2logstash.py", line 194, in start
dmarc2filebeat        |     download(server, username, password, jsonOutputFile, timeout, shouldDelete, shouldDeleteFailures)
dmarc2filebeat        |   File "/opt/dmarc2logstash/dmarc2logstash.py", line 47, in download
dmarc2filebeat        |     conn = connect(server, username, password, timeout)
dmarc2filebeat        |   File "/opt/dmarc2logstash/dmarc2logstash.py", line 32, in connect
dmarc2filebeat        |     conn = poplib.POP3_SSL(server)
dmarc2filebeat        |   File "/usr/lib/python3.10/poplib.py", line 452, in __init__
dmarc2filebeat        |     POP3.__init__(self, host, port, timeout)
dmarc2filebeat        |   File "/usr/lib/python3.10/poplib.py", line 104, in __init__
dmarc2filebeat        |     self.sock = self._create_socket(timeout)
dmarc2filebeat        |   File "/usr/lib/python3.10/poplib.py", line 455, in _create_socket
dmarc2filebeat        |     sock = POP3._create_socket(self, timeout)
dmarc2filebeat        |   File "/usr/lib/python3.10/poplib.py", line 112, in _create_socket
dmarc2filebeat        |     return socket.create_connection((self.host, self.port), timeout)
dmarc2filebeat        |   File "/usr/lib/python3.10/socket.py", line 824, in create_connection
dmarc2filebeat        |     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
dmarc2filebeat        |   File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
dmarc2filebeat        |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
dmarc2filebeat        | socket.gaierror: [Errno -3] Try again
dmarc2filebeat exited with code 1

The docker-compose is built like this:

  dmarc2filebeat:
    image: jertel/dmarc2logstash:latest
    container_name: dmarc2filebeat
    mem_limit: 1073741824
    volumes:
      - ./dmarc/logs:/logs
      - ./dmarc/dmarc2logstash.json:/opt/dmarc2logstash/dmarc2logstash.json
      - ./dmarc/dmarclogs:/dmarclogs
    environment:
      - POP3_SERVER=popserver.dk
      - POP3_USERNAME=dmarc@popserver.dk
      - POP3_PASSWORD=verysecret
      - DELETE_MESSAGES=1
      - JSON_OUTPUT_FILE=/dmarclogs/dmarc.json
      - DELETE_FAILURES=1
      - SLEEP_SECONDS=600
      - SOCKET_TIMEOUT_SECONDS=10
    restart: always
    depends_on:
      - filebeat_for_dmarc

I can ping the popserver, and also telnet to port 110 from the host. Any ideas on how to proceed?

[jertel]

I'm not sure what error -3 means. Check that your POP3_SERVER value is correct and doesn't have extra spaces, quotes, etc, and that you can resolve the DNS hostname, and that it's accessible (via ping/telnet) from within that container -- Not just from the host. You will need to exec into the running dmarc2logstash container to verify this.

[fribse]

Hi @jertel I can't exec into the container, as it crashes all the time. I've gone ahead and done the painfull thing of deleting the messages with the reports, that didn't help at all. I've also tried deleting the old concated log files.

[jertel]

It seems to be a docker DNS resolution issue so you could try spinning up a temporary container to run your ping test.

docker run --rm --entrypoint bash -it jertel/dmarc2logstash

Then run the ping test against your popserver host.

[fribse]

Hi again

I managed to catch it while it was up

[root@alpha00035 docker]# docker exec -it dmarc2filebeat /bin/sh
/opt/dmarc2logstash # ping bounce.xxxxx.dk
PING bounce.XXXXX.dk (192.168.3.22): 56 data bytes
64 bytes from 192.168.3.22: seq=0 ttl=62 time=0.543 ms
64 bytes from 192.168.3.22: seq=1 ttl=62 time=0.490 ms

In the docker-compose it reads:

  dmarc2filebeat:
    image: jertel/dmarc2logstash:latest
    container_name: dmarc2filebeat
    mem_limit: 1073741824
    volumes:
      - ./dmarc/logs:/logs
      - ./dmarc/dmarc2logstash.json:/opt/dmarc2logstash/dmarc2logstash.json
      - ./dmarc/dmarclogs:/dmarclogs
    environment:
      - POP3_SERVER=bounce.XXX.dk
      - POP3_USERNAME=dmarc@XXX.dk
      - POP3_PASSWORD=XXX
      - DELETE_MESSAGES=1
      - JSON_OUTPUT_FILE=/dmarclogs/dmarc.json
      - DELETE_FAILURES=1
      - SLEEP_SECONDS=600
      - SOCKET_TIMEOUT_SECONDS=10
    restart: always
    depends_on:
      - filebeat_for_dmarc

[jertel]

My next debugging steps would be to hardcode the DNS into the container. Add this to your docker-compose:

extra_hosts:
  - "popserver.dk:192.168.3.22"

Replacing the popserver.dk with your actual POP3 hostname.

Another debugging step is to try hardcoding the POP3 server IP directly into the POP3_SERVER variable.

[fribse]

Good idea, I changed the variable to use the IP directly, and now I see it scroll through messages, not sure what to grep for in the logfile to see the connection command...

[jertel]

There are a few docker DNS related parameters you can play with, if you want to avoid hardcoding the IP long-term. Look at the Docker documentation and perhaps try using an explicit DNS server inside the container.

[fribse]

I made it work as it should now, thankyou for your help!