search

Graylog2 / graylog-docker

Official Graylog Docker image
https://hub.docker.com/r/graylog/graylog/
Apache License 2.0
372 stars 134 forks source link

doenst process messages #89

Closed saschaarthur closed 5 years ago

saschaarthur commented 5 years ago

Hello

Graylog regulary doenst process messages for me. Im currently on 3.1.1-1. Elasticsearch cluster is green, journal is growing (in > 300, out = 0). Mongodb green and reachable.

1

on the stdout of the container i found

stack-1.txt and stack-2.txt

heres the threadump: threads.txt

its the worst what could happen to graylog because it makes it pretty useless.. yes i cleaned the journal as well, comes from time to time back please advice.

saschaarthur commented 5 years ago

recreating/closing the indices does not work as well..

florianpopp commented 5 years ago

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.

Thank you!

saschaarthur commented 5 years ago

ehm could you explain what exactly should be the cause when not graylog itself?

Everything except graylog is working without trouble (docker,mongo, elastic) just graylog is stuck processing things.

Just closing this an saying its not graylogs issue is more then wrong IMO.

especially because stack-1 shows that its deply crashing in kafka / graylog..

saschaarthur commented 5 years ago

And i have already a guess whats the problem, need to leave it running a bit longer until i see if its solved now..

Update: After a couple of minutes / hours graylog also jumps from processing to paused and LB status is DEAD/Throttled

Im using since ages a mapped filesystem location (ext4) for the journal (not a docker volume). I switched it yesterday to docker-volume, since now its running.

Could mapping a FS location break graylog from processing messages?

jalogisch commented 5 years ago

@saschaarthur as @florianvolle wrote - this is not a Graylog issue it is a configuration issue on your installation. That is why he points you to the community board - as this is the place where we can discuss and check.

For me - from the first stack, it look like your journal is corrupt. But as you did not provide any information on your setup and configuration, we can only point you to the place where we discuss.

So please move over to the place where we will actually try to help you - because the image is out quite a time plus we use that our self.

saschaarthur commented 5 years ago

Im using the docker-compose example from the latest 3.1 documentation (https://docs.graylog.org/en/3.1/pages/installation/docker.html).

because the image is out quite a time plus we use that our self.

Thats nice, but not really a good reason why graylog could not be stuck in exactly this state, its just lowering the chance.

Btw this 'stuck' processing i had i older graylog versions as well (on different servers, differnt os, different docker versions, for sure), but i was to lazy to report it until now, the reason why i overcome it now is that in the new version it happens much more often..

Anyways for gods sake: https://community.graylog.org/t/graylog-not-processing-messages/11902