[Bug]: MondoDB not starting

[travisfranck]

Controller Version

v5.9

Describe the Bug

I've googled around here and the TPLink forums. I'm not sure where the problems lies, but I'm wondering if the MongoDB got corrupted like one of your warnings suggests.

I've been using this image and upgrading for 1.5 years. the image stopped working, I think when I upgraded to v5.9. (feb?). The server.log is included below.

I assume that the v5.9 image includes all the Mongo requirements, and if it didn't it would have been noticed by more people.

I'm not super familiar with Mongo so I was about to wipe the enter docker installation and have to reset up my network. Is there something I can try before this?

Thank you in advance.

Expected Behavior

Image starting successfully.

Steps to Reproduce

1. Download latest image 5.9.x
2. Use the docker-compose example from the website (though before my docker-compose didn't have the PUID/PGID lines)
3. docker-compose up -d

How You're Launching the Container

Docker-compose

Container Logs

pi@rpi:~/Docker $ sudo tail -20 /var/lib/docker/volumes/docker_omada-logs/_data/server.log
04-19-2023 10:35:27.547 INFO [main] [] c.t.s.o.s.OmadaLinuxMain(): record: start the omada controller
04-19-2023 10:35:27.567 INFO [main] [] c.t.s.o.s.OmadaLinuxMain(): record: set property finished
04-19-2023 10:35:27.579 INFO [main] [] c.t.s.o.s.OmadaLinuxMain(): record: configure log finished
04-19-2023 10:35:27.591 INFO [main] [] c.t.s.o.s.OmadaBootstrap(): record: bootstrap prepare
04-19-2023 10:35:27.600 INFO [log4j-thread] [] c.t.s.o.c.o.a.b(): success to load configuration omada.properties
04-19-2023 10:35:27.604 INFO [log4j-thread] [] c.t.s.o.c.o.OmadacType(): omadacType: Local Controller
04-19-2023 10:35:27.672 INFO [main] [] c.t.s.o.s.OmadaBootstrap(): going to start local mongod.
04-19-2023 10:37:07.891 ERROR [main] [] c.t.s.f.c.FacadeUtils(): facadeMsgEnable is not enable, msg: Fail to start mongo DB server
04-19-2023 10:37:07.891 WARN [main] [] c.t.s.o.s.s.b(): Fail to start mongo DB server
04-19-2023 10:37:07.891 WARN [main] [] c.t.s.o.s.s.a(): com.tplink.smb.omada.starter.b.a: Fail to start mongo DB server
com.tplink.smb.omada.starter.b.a: Fail to start mongo DB server
    at com.tplink.smb.omada.starter.server.b.a(SourceFile:151) ~[local-starter-5.9.31.jar:5.9.31]
    at com.tplink.smb.omada.starter.server.a.e(SourceFile:74) ~[local-starter-5.9.31.jar:5.9.31]
    at com.tplink.smb.omada.starter.OmadaBootstrap.d(SourceFile:249) ~[local-starter-5.9.31.jar:5.9.31]
    at com.tplink.smb.omada.starter.OmadaBootstrap.q(SourceFile:329) ~[local-starter-5.9.31.jar:5.9.31]
    at com.tplink.smb.omada.starter.OmadaBootstrap.a(SourceFile:116) ~[local-starter-5.9.31.jar:5.9.31]
    at com.tplink.smb.omada.starter.OmadaBootstrap.e(SourceFile:278) ~[local-starter-5.9.31.jar:5.9.31]
    at com.tplink.smb.omada.starter.OmadaLinuxMain.a(SourceFile:92) ~[local-starter-5.9.31.jar:5.9.31]
    at com.tplink.smb.omada.starter.OmadaLinuxMain.main(SourceFile:42) ~[local-starter-5.9.31.jar:5.9.31]
pi@rpi:~/Docker $ 

### Additional Context

_No response_

[mbentley]

Could you get the MongoDB logs from mongod.log which is in the same folder as the server.log file from the logs volume?

I am adding a compose option to allow for clean shutdown default timeouts in https://github.com/mbentley/docker-omada-controller/pull/303/commits/4bac3f9c72f16ddd200902a3eb0c4ab5bcb385ad as well as notes in the readme about clean shutdowns. In general and especially on a device like a raspberry pi, the default stop timeout will not allow for the container + MongoDB to shut down cleanly in the default 10 seconds.

[mbentley]

Sorry, the note in the PR closed this prematurely.

[travisfranck]

Again, thank you in advance for your help.

Here are the last 20 lines of the mongod.log:

pi@rpi:~ $ sudo tail -20 /var/lib/docker/volumes/docker_omada-logs/_data/mongod.log
 /opt/tplink/EAPController/bin/mongod(_ZN5mongo21runGlobalInitializersEiPKPKcS3_+0x22b) [0x1fcdc8]
 /opt/tplink/EAPController/bin/mongod(main+0x6f) [0x1c4e64]
 /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x9d) [0xf71a38aa]
2023-04-19T10:40:54.998-0400 ***** SERVER RESTARTED *****
2023-04-19T10:40:55.000-0400 SEVERE: Invalid access at address: 0x49d333e
2023-04-19T10:40:55.004-0400 SEVERE: Got signal: 7 (Bus error).
Backtrace:0x7639be 0x762f6e 0x76300a 0xf703c270 0x75ece4 0x75cd7e 0x75cdf4 0x1f0300 0x1fc81a 0x1fcb7e 0x1fcdc8 0x1c4e64 0xf702d8aa 
 /opt/tplink/EAPController/bin/mongod(_ZN5mongo15printStackTraceERSo+0x25) [0x7639be]
 /opt/tplink/EAPController/bin/mongod() [0x762f6e]
 /opt/tplink/EAPController/bin/mongod() [0x76300a]
 /lib/arm-linux-gnueabihf/libc.so.6(+0x25270) [0xf703c270]
 /opt/tplink/EAPController/bin/mongod(_ZN5mongo11ProcessInfo10SystemInfo17collectSystemInfoEv+0x87f) [0x75ece4]
 /opt/tplink/EAPController/bin/mongod(_ZN5mongo11ProcessInfo20initializeSystemInfoEv+0x7d) [0x75cd7e]
 /opt/tplink/EAPController/bin/mongod(_ZN5mongo36_mongoInitializerFunction_SystemInfoEPNS_18InitializerContextE+0x7) [0x75cdf4]
 /opt/tplink/EAPController/bin/mongod(_ZN5boost6detail8function17function_invoker1IPFN5mongo6StatusEPNS3_18InitializerContextEES4_S6_E6invokeERNS1_15function_bufferES6_+0x17) [0x1f0300]
 /opt/tplink/EAPController/bin/mongod(_ZNK5mongo11Initializer7executeERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EERKSt3mapIS7_S7_St4lessIS7_ESaISt4pairIKS7_S7_EEE+0x15d) [0x1fc81a]
 /opt/tplink/EAPController/bin/mongod(_ZN5mongo21runGlobalInitializersERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS6_EERKSt3mapIS6_S6_St4lessIS6_ESaISt4pairIKS6_S6_EEE+0x29) [0x1fcb7e]
 /opt/tplink/EAPController/bin/mongod(_ZN5mongo21runGlobalInitializersEiPKPKcS3_+0x22b) [0x1fcdc8]
 /opt/tplink/EAPController/bin/mongod(main+0x6f) [0x1c4e64]
 /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x9d) [0xf702d8aa]
pi@rpi:~ $ 

[travisfranck]

Oh, and for the record, I'm running this on a Raspberry Pi 4, which I think you already surmised.

[mbentley]

Could you share the output from a few commands:

• uname -a
• docker info --format '{{json .OSType}} {{json .Architecture}}'
• docker images --filter=reference='mbentley/omada-controller' --digests

I want to be able verify what kernel you're running to see if your Pi 4 is running a 32 or 64 bit OS, same with what Docker reports and I want to see what specific image you're running. It looks like from the errors that you're running a 32 bit OS based on the /lib path mentioned for libc.so.6 which means you're likely on the sketchy version of MongoDB due to the lack of continued 32 bit support but that being said, I don't know exactly what is happening there as I am not sure I've seen this error from MongoDB before.

[travisfranck]

Here you go:

pi@rpi:~ $ uname -a
Linux rpi 6.1.19-v8+ #1637 SMP PREEMPT Tue Mar 14 11:11:47 GMT 2023 aarch64 GNU/Linux
pi@rpi:~ $ uname -a
Linux rpi 6.1.19-v8+ #1637 SMP PREEMPT Tue Mar 14 11:11:47 GMT 2023 aarch64 GNU/Linux
pi@rpi:~ $ docker info --format '{{json .OSType}} {{json .Architecture}}'
"linux" "aarch64"
pi@rpi:~ $ docker images --filter=reference='mbentley/omada-controller' --digests
REPOSITORY                  TAG       DIGEST                                                                    IMAGE ID       CREATED         SIZE
mbentley/omada-controller   5.9       sha256:0043c77858d5ac7a231d5504aa01d569ea01544d6b82bebb394e03bc9796356a   6aa02aa7eccd   3 weeks ago     565MB
mbentley/omada-controller   5.9       sha256:0f8b7b3b93eaefee8b5114b81e0d732c11500d5da770f06dd605fd676787d9d4   6aa02aa7eccd   3 weeks ago     565MB
mbentley/omada-controller   latest    sha256:0043c77858d5ac7a231d5504aa01d569ea01544d6b82bebb394e03bc9796356a   6aa02aa7eccd   3 weeks ago     565MB
mbentley/omada-controller   latest    sha256:0f8b7b3b93eaefee8b5114b81e0d732c11500d5da770f06dd605fd676787d9d4   6aa02aa7eccd   3 weeks ago     565MB
mbentley/omada-controller   <none>    sha256:b81a151d6f4cd96dccf49c06b75b0c257997b3b56b06ca6cdb93eeb659f29174   a04f8d23ef18   4 weeks ago     558MB
mbentley/omada-controller   <none>    sha256:dbc036862e61b356c44f01f9b9863777dab4da16656556004568e0894c8b5718   85200560093f   7 weeks ago     558MB
mbentley/omada-controller   <none>    sha256:1fa9c009cb708247a632ab0b88f7587aad7c8e9c1e3d1953099ed17e1c52ed03   199e6f095c2f   2 months ago    557MB
mbentley/omada-controller   <none>    sha256:18ac8f39889fc3446540a0aee45632e447a84a00134f1bd6a401bf72fd87bf29   0f41b77372ac   4 months ago    575MB
mbentley/omada-controller   <none>    sha256:5d99085482c69c050cbe4ce5893a2733044b593269563749d4c848ae4190f34a   2c29ab7bd278   4 months ago    575MB
mbentley/omada-controller   <none>    sha256:c21ec9a66f1676ff2db7cb545b095d7f4f84e10f9285d987b694b17bbf0ccfe3   15978a83bd69   5 months ago    545MB
mbentley/omada-controller   <none>    sha256:765290f355955dd9f1cc14c928d9bd22b2e9f22e72020b95a6e93a1f821e7a95   f8fc7f63e0e7   8 months ago    542MB
mbentley/omada-controller   <none>    sha256:463d6de0f06c3546416bc4f8bc5ebce44852d12ae18bbdd6fcdf4908cc71c868   16ea5f9cb7f8   11 months ago   538MB
mbentley/omada-controller   <none>    sha256:118db0e0dff6ec7f47eea2c4e156a7d0aa9ad8c812cecfea2069f1be3528ff10   079d9ad2110e   12 months ago   535MB
mbentley/omada-controller   <none>    sha256:cb82dc0d2c3a50b6e3b768c68392bb3644138d3f61e900975eb8e83bbb05b367   ac6af8d2cc19   13 months ago   468MB

I'll add that I did update my packages on the rPi, so maybe that is the part of the environment that changed. Maybe the new packages aren't compatible or got messed up. I guess I thought docker containers are supposed to be self-contained, so I didn't think upgrading PiOS would cause problems.

[mbentley]

OK, good news - so I can reproduce the issue. If I try to start the linux/arm/v7 image on system that is running a 64 bit kernel, it fails to start.

I am not sure why it would pull the linux/arm/v7 image over the linux/amd64 but it did or you were previously running a 32 bit kernel.

You can specify pulling the linux/arm64 image when pulling:

docker pull --platform linux/arm64 mbentley/omada-controller:5.9

I don't see a specific platform option in the Docker Compose spec but I think you could set this env var to make it work optionally if it is defaulting to linux/arm/v7: export DOCKER_DEFAULT_PLATFORM=linux/amd64

Give that a shot.

[mbentley]

OK, I think I figured out what is going on. See https://github.com/mbentley/docker-omada-controller/issues/304#issuecomment-1531299402.

[travisfranck]

Last week I tried what you suggested above and it didn't work make any difference that I could tell. I wasn't sure so I didn't post anything because I wanted to be more thorough. Apologies in the delay.

I posted on the other issue thread. Happy to follow that thread it you think we can consolidate.

[mbentley]

Closing to consolidate conversation in #304.