mongodb errors in systemlog

[rogern1970]

(mongod), uid 0: exited on signal 6 (core dumped) is what i see and unifi is looping forever. It never starts.

[jchillerup]

I have this problem, too.

pid 4358 (mongod), uid 0: exited on signal 6 (core dumped)
pid 65053 (mongod), uid 0: exited on signal 6 (core dumped)
pid 83863 (mongod), uid 0: exited on signal 6 (core dumped)
pid 41030 (mongod), uid 0: exited on signal 6 (core dumped)
[HBSD SEGVGUARD] [mongod (41030)] Suspending execution for 600 seconds after 5 crashes.
 -> pid: 41030 ppid: 4586 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT> 
[HBSD SEGVGUARD] [bin/mongod (65197)] Preventing execution due to repeated segfaults.
 -> pid: 65197 ppid: 4586 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT> 
[HBSD SEGVGUARD] [bin/mongod (67107)] Preventing execution due to repeated segfaults.
 -> pid: 67107 ppid: 4586 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT> 
[HBSD SEGVGUARD] [bin/mongod (13646)] Preventing execution due to repeated segfaults.
 -> pid: 13646 ppid: 4586 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT> 
[HBSD SEGVGUARD] [bin/mongod (63348)] Preventing execution due to repeated segfaults.
 -> pid: 63348 ppid: 4586 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>```

[ztripez]

I have the same problem

[jmbwell]

Memory constraints, perhaps? Or perhaps are you on a 32-bit system?

[RipNRun]

I'm having the exact same problems. I'm on a 64-bit Netgate system, so I don't think memory is the issue.

[court-bailey]

Same problem here reinstall pfsense and unifi a couple of times now. Sometimes it happens in hours, sometimes days. But eventually just starts flooding the logs with the mongo error and the unifi controller never starts. seems the mongodb is getting corrupted some way.

64bit install/ 16GB of ram whould think that should be fine.

[dustinduse]

Did no one look at the mongo logs?

2019-04-25T21:38:57.403+0600 I CONTROL  [main] ***** SERVER RESTARTED *****
2019-04-25T21:38:57.414+0600 I CONTROL  [initandlisten] MongoDB starting : pid=81733 port=27117 dbpath=/usr/local/UniFi/data/db 64-bit host=CTIPF1.computerteam.com
2019-04-25T21:38:57.415+0600 I CONTROL  [initandlisten] db version v3.4.20
2019-04-25T21:38:57.415+0600 I CONTROL  [initandlisten] git version: 447847d93d6e0a21b018d5df45528e815c7c13d8
2019-04-25T21:38:57.415+0600 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2o-freebsd  27 Mar 2018
2019-04-25T21:38:57.415+0600 I CONTROL  [initandlisten] allocator: system
2019-04-25T21:38:57.415+0600 I CONTROL  [initandlisten] modules: none
2019-04-25T21:38:57.415+0600 I CONTROL  [initandlisten] build environment:
2019-04-25T21:38:57.415+0600 I CONTROL  [initandlisten]     distarch: x86_64
2019-04-25T21:38:57.415+0600 I CONTROL  [initandlisten]     target_arch: x86_64
2019-04-25T21:38:57.415+0600 I CONTROL  [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 27117, unixDomainSocket: { pathPrefix: "/usr/local/UniFi/run" } }, storage: { dbPath: "/usr/local/UniFi/data/db" }, systemLog: { destination: "file", logAppend: true, path: "/usr/local/UniFi/logs/mongod.log" } }
2019-04-25T21:38:57.415+0600 I -        [initandlisten] Detected data files in /usr/local/UniFi/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-04-25T21:38:57.415+0600 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3276M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2019-04-25T21:38:57.419+0600 E STORAGE  [initandlisten] WiredTiger error (0) [1556206737:419171][81733:0x808c16000], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2019-04-25T21:38:57.419+0600 E STORAGE  [initandlisten] WiredTiger error (-31804) [1556206737:419317][81733:0x808c16000], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-04-25T21:38:57.419+0600 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365
2019-04-25T21:38:57.419+0600 I -        [initandlisten]

***aborting after fassert() failure

2019-04-25T21:38:57.420+0600 F -        [initandlisten] Got signal: 6 (Abort trap).

`

[kernsche]

We also have the same problem with 2 controllers.

2019-08-02T11:06:39.462+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2019-08-02T11:06:39.491+0200 I CONTROL  [initandlisten] MongoDB starting : pid=83291 port=27117 dbpath=/usr/local/UniFi/data/db 64-bit host=S44Pfsense.S44.local
2019-08-02T11:06:39.491+0200 I CONTROL  [initandlisten] db version v3.4.21
2019-08-02T11:06:39.491+0200 I CONTROL  [initandlisten] git version: ba2e818de44c9a331ce09a624780a2bd841cf6e4
2019-08-02T11:06:39.491+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2o-freebsd  27 Mar 2018
2019-08-02T11:06:39.491+0200 I CONTROL  [initandlisten] allocator: system
2019-08-02T11:06:39.491+0200 I CONTROL  [initandlisten] modules: none
2019-08-02T11:06:39.491+0200 I CONTROL  [initandlisten] build environment:
2019-08-02T11:06:39.491+0200 I CONTROL  [initandlisten]     distarch: x86_64
2019-08-02T11:06:39.491+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2019-08-02T11:06:39.491+0200 I CONTROL  [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 27117, unixDomainSocket: { pathPrefix: "/usr/local/UniFi/run" } }, storage: { dbPath: "/usr/local/UniFi/data/db" }, systemLog: { destination: "file", logAppend: true, path: "/usr/local/UniFi/logs/mongod.log" } }
2019-08-02T11:06:39.492+0200 I -        [initandlisten] Detected data files in /usr/local/UniFi/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-08-02T11:06:39.493+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3559M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2019-08-02T11:06:39.499+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1564736799:499929][83291:0x808c16000], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2019-08-02T11:06:39.499+0200 E STORAGE  [initandlisten] WiredTiger error (-31804) [1564736799:499957][83291:0x808c16000], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-08-02T11:06:39.499+0200 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365
2019-08-02T11:06:39.499+0200 I -        [initandlisten]

***aborting after fassert() failure

2019-08-02T11:06:39.503+0200 F -        [initandlisten] Got signal: 6 (Abort trap).

 0x2316349 0x2315b7b 0x8071b5946 0x8071b4eb2
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"1021000","o":"12F5349","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"},{"b":"1021000","o":"12F4B7B","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"8071A7000","o":"E946","s":"pthread_sigmask"},{"b":"8071A7000","o":"DEB2","s":"pthread_getspecific"}],"processInfo":{ "mongodbVersion" : "3.4.21", "gitVersion" : "ba2e818de44c9a331ce09a624780a2bd841cf6e4", "compiledModules" : [], "uname" : { "sysname" : "FreeBSD", "release" : "11.2-RELEASE-p10", "version" : "FreeBSD 11.2-RELEASE-p10 #9 4a2bfdce133(RELENG_2_4_4): Wed May 15 18:54:42 EDT 2019     root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense", "machine" : "amd64" } }}
 mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x2316349]
 mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x15B) [0x2315b7b]
 libthr.so.3(pthread_sigmask+0x536) [0x8071b5946]
 libthr.so.3(pthread_getspecific+0xE12) [0x8071b4eb2]
-----  END BACKTRACE  -----

We reinstalled the Controller mutliple times, but its the same thing @court-bailey wrote:

Same problem here reinstall pfsense and unifi a couple of times now. Sometimes it happens in hours, sometimes days. But eventually just starts flooding the logs with the mongo error and the unifi controller never starts. seems the mongodb is getting corrupted some way.

[NightKhaos]

I've also encountered this error quite often. It appears to happen after a non-clean shutdown.

As far as I can tell, mongod --repair should be able to fix these kind of errors, but only if the fixes have been installed. According to the repair documentation, that only applies to Mongo DB 4.0.3 and later.

Therefore the ultimate fix will be to upgrade to Mongo DB 4.0.3 or later. We should see see if I can work out how to install MongoDB 4 since that sounds like the best solution, otherwise we'll need to work out how to backport the fix.

As the latest version of the UniFi controller STILL only supports Mongo DB 3.4 despite it going EOL, we either need to wait until fall when UniFi should be updated to support a later version of Mongo or try our hand at backporting.

[cc12258]

I was getting the same mongod error repeating in the log and was able to repair it by doing this:

service unifi.sh stop
service mongod stop

Check to make sure mongo stopped, in my case using the service call didn't work so had to manually kill it:

ps -aux | grep mongo
kill -9 [pid]

Then repair the database:

mongod --dbpath /usr/local/UniFi/data/db --repair
service mongod start

Afterwards you can restart the unifi service:

service unifi.sh start

[gnkidwell]

This issue has been resolved with mongodb repair script in the current install script. Closing this for now.