search

Graylog2 / graylog2-images

Ready to run machine images
Apache License 2.0
238 stars 91 forks source link

My Graylog server Mongodb is Down #223

Closed momaydopod closed 5 years ago

momaydopod commented 5 years ago

My Graylog server Mongodb is Down. how to resolve ?

LOG01:/dev# graylog-ctl status run: elasticsearch: (pid 20257) 0s; run: log: (pid 846) 7239s run: etcd: (pid 6982) 421s; run: log: (pid 845) 7239s run: graylog-server: (pid 6996) 421s; run: log: (pid 847) 7239s down: mongodb: 0s, normally up, want up; run: log: (pid 843) 7239s run: nginx: (pid 7097) 419s; run: log: (pid 844) 7239s

`Graylog is restarting... There is no Graylog web application running at the moment, please reload this page in a minute. It can take up to 1-2 minutes until all services are running properly. In case this is a permanent error, check the following:

Check if all services are running - sudo graylog-ctl status shows an overview of all running services Check for errors in log files - Relevant services write log files here: /var/log/graylog/*/current Ask for help - If there is no way to fix the issue ask for help: https://github.com/Graylog2/graylog2-images/issues`

2019-01-02_09:01:04.37305 2019-01-02T09:01:04.369+0000 W - [initandlisten] Detected unclean shutdown - /var/opt/graylog/data/mongodb/mongod.lock is not empty. 2019-01-02_09:01:05.48589 2019-01-02T09:01:05.476+0000 W - [initandlisten] Detected unclean shutdown - /var/opt/graylog/data/mongodb/mongod.lock is not empty. 2019-01-02_09:01:06.66207 2019-01-02T09:01:06.655+0000 W - [initandlisten] Detected unclean shutdown - /var/opt/graylog/data/mongodb/mongod.lock is not empty. 2019-01-02_09:01:07.77708 2019-01-02T09:01:07.772+0000 W - [initandlisten] Detected unclean shutdown - /var/opt/graylog/data/mongodb/mongod.lock is not empty. 2019-01-02_09:01:08.89388 2019-01-02T09:01:08.887+0000 W - [initandlisten] Detected unclean shutdown - /var/opt/graylog/data/mongodb/mongod.lock is not empty. 2019-01-02_09:01:10.00927 2019-01-02T09:01:10.004+0000 W - [initandlisten] Detected unclean shutdown - /var/opt/graylog/data/mongodb/mongod.lock is not empty. 2019-01-02_09:01:11.13367 2019-01-02T09:01:11.123+0000 W - [initandlisten] Detected unclean shutdown - /var/opt/graylog/data/mongodb/mongod.lock is not empty. 2019-01-02_09:01:12.23550 2019-01-02T09:01:12.229+0000 W - [initandlisten] Detected unclean shutdown - /var/opt/graylog/data/mongodb/mongod.lock is not empty. 2019-01-02_09:01:13.37530 2019-01-02T09:01:13.373+0000 W - [initandlisten] Detected unclean shutdown - /var/opt/graylog/data/mongodb/mongod.lock is not empty.

mariussturm commented 5 years ago

Hi, looks like the VM was stopped without a clean shutdown. Check that the system is in a decent state, no filesystem problems, enough disk space free... Then remove the old lock-file /var/opt/graylog/data/mongodb/mongod.lock and restart MongoDB.

momaydopod commented 5 years ago

Hi, looks like the VM was stopped without a clean shutdown. Check that the system is in a decent state, no filesystem problems, enough disk space free... Then remove the old lock-file /var/opt/graylog/data/mongodb/mongod.lock and restart MongoDB.

I try rm -rf /var/opt/graylog/data/mongodb/mongod.lock and then restart server is not wok

momaydopod commented 5 years ago

My disk

:/home/ubuntu# df Filesystem 1K-blocks Used Available Use% Mounted on udev 2009296 4 2009292 1% /dev tmpfs 404656 440 404216 1% /run /dev/dm-0 15648688 15121812 0 100% / none 4 0 4 0% /sys/fs/cgroup none 5120 0 5120 0% /run/lock none 2023272 0 2023272 0% /run/shm none 102400 0 102400 0% /run/user /dev/sda1 240972 79185 149346 35% /boot overflow 1024 32 992 4% /tmp

momaydopod commented 5 years ago

I add new disk to VM and follow this

http://docs.graylog.org/en/2.2/pages/configuration/graylog_ctl.html#extend-ova-disk