Stuck At Upgrading Screen

[NathanTheGr8]

I have two docker containers that broke when upgrading recently. I am not sure when it happened, maybe last week, but I noticed it yesterday. Looking at the docker logs I see this

|   "-Xmx32240M" (0x00000000),
|   "-XX:+HeapDumpOnOutOfMemoryError" (0x00000000),
|   "-XX:+UseG1GC" (0x00000000),
|   "-XX:+UseStringDeduplication" (0x00000000),
|   "-Djava.library.path=/usr/lib/unifi-video/lib" (0x00000000),
|   "-Djava.awt.headless=true" (0x00000000),
|   "-Djavax.net.ssl.trustStore=/usr/lib/unifi-video/data/ufv-truststore" (0x00000000),
|   "-Dfile.encoding=UTF-8" (0x00000000),
|   "-Dcom.sun.management.jmxremote" (0x00000000),
|   "-Dcom.sun.management.jmxremote.ssl=false" (0x00000000),
|   "-Dcom.sun.management.jmxremote.authenticate=false" (0x00000000),
|   "-Dcom.sun.management.jmxremote.port=7654" (0x00000000),
|   "-Djava.rmi.server.hostname=172.17.0.6" (0x00000000),
+-------------------------------------------------------,
| Internal options:              4,
|   "-Dcommons.daemon.process.id=25324" (0x00000000),
|   "-Dcommons.daemon.process.parent=111" (0x00000000),
|   "-Dcommons.daemon.version=1.0.15-dev" (0x00000000),
|   "abort" (0xbaeb94e0),
+-------------------------------------------------------,
Java VM created successfully,
Class org/apache/commons/daemon/support/DaemonLoader found,
Native methods registered,
java_init done,
Daemon loading...,
.2019-02-27 22:20:26,395 ERROR Unable to locate appender ConsoleAppender for logger ,
emon loaded successfully,
java_load done,
dropped capabilities.,
..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................#,
# There is insufficient memory for the Java Runtime Environment to continue.,
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.,
# An error report file with more information is saved as:,
# /usr/lib/unifi-video/hs_err_pid25324.log,
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f0d982c3000, 12288, 0) failed; error='Cannot allocate memory' (errno=12),
OpenJDK 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.,
Reloading service,
Attemtping to load library /lib64/libcap.so.2,
Attemtping to load library /lib64/libcap.so.1,
Attemtping to load library /lib64/libcap.so,
Attemtping to load library /usr/lib64/libcap.so.2,
Attemtping to load library /usr/lib64/libcap.so.1,
Attemtping to load library /usr/lib64/libcap.so,
Attemtping to load library /lib/libcap.so.2,
Attemtping to load library /lib/libcap.so.1,
Attemtping to load library /lib/libcap.so,
Attemtping to load library /usr/lib/libcap.so.2,
Attemtping to load library /usr/lib/libcap.so.1,
Attemtping to load library /usr/lib/libcap.so,
Attemtping to load library libcap.so.2,
loaded cap_free from libcap.,
loaded cap_init from libcap.,
loaded cap_clear from libcap.,
loaded cap_get_flag from libcap.,
loaded cap_set_flag from libcap.,
loaded cap_set_proc from libcap.,
increased capability set.,
user changed to 'unifi-video',
decreased capability set to min required.,
Using default JVM in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so,
Attemtping to load library /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so,
JVM library /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so loaded,
JVM library entry point found (0x22D601E0),
+-- DUMPING JAVA VM CREATION ARGUMENTS -----------------,
| Version:                       0x010004,
| Ignore Unrecognized Arguments: True,
| Extra options:                 17,
|   "-Djava.class.path=/usr/share/java/commons-daemon.jar:/usr/lib/unifi-video/lib/airvision.jar" (0x00000000),
|   "-Dav.tempdir=/var/cache/unifi-video" (0x00000000),
|   "-Djava.security.egd=file:/dev/./urandom" (0x00000000),
|   "-Xms32240M" (0x00000000),
|   "-Xmx32240M" (0x00000000),
|   "-XX:+HeapDumpOnOutOfMemoryError" (0x00000000),
|   "-XX:+UseG1GC" (0x00000000),
|   "-XX:+UseStringDeduplication" (0x00000000),
|   "-Djava.library.path=/usr/lib/unifi-video/lib" (0x00000000),
|   "-Djava.awt.headless=true" (0x00000000),
|   "-Djavax.net.ssl.trustStore=/usr/lib/unifi-video/data/ufv-truststore" (0x00000000),
|   "-Dfile.encoding=UTF-8" (0x00000000),
|   "-Dcom.sun.management.jmxremote" (0x00000000),
|   "-Dcom.sun.management.jmxremote.ssl=false" (0x00000000),
|   "-Dcom.sun.management.jmxremote.authenticate=false" (0x00000000),
|   "-Dcom.sun.management.jmxremote.port=7654" (0x00000000),
|   "-Djava.rmi.server.hostname=172.17.0.6" (0x00000000),
+-------------------------------------------------------,
| Internal options:              4,
|   "-Dcommons.daemon.process.id=23564" (0x00000000),
|   "-Dcommons.daemon.process.parent=111" (0x00000000),
|   "-Dcommons.daemon.version=1.0.15-dev" (0x00000000),
|   "abort" (0xbaeb94e0),
+-------------------------------------------------------,
Java VM created successfully,
Class org/apache/commons/daemon/support/DaemonLoader found,
Native methods registered,
java_init done,
Daemon loading...,
.2019-02-27 22:42:24,784 ERROR Unable to locate appender ConsoleAppender for logger ,
emon loaded successfully,
java_load done,
dropped capabilities.,

So some error about java running out of resources? The system has and had plenty of free ram. I wasn't running the docker container with any sort of constraits. I recreated the container and got this error.

|   "start",
+-------------------------------------------------------,
user changed to 'unifi-video',
User 'unifi-video' validated,
Attempting to locate Java Home in /usr/lib/jvm/java-8-openjdk-amd64/jre,
Attempting to locate VM configuration file /usr/lib/jvm/java-8-openjdk-amd64/jre/jre/lib/jvm.cfg,
Attempting to locate VM configuration file /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jvm.cfg,
Attempting to locate VM configuration file /usr/lib/jvm/java-8-openjdk-amd64/jre/jre/lib/amd64/jvm.cfg,
Attempting to locate VM configuration file /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/jvm.cfg,
Found VM configuration file at /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/jvm.cfg,
Found VM server definition in configuration,
Checking library /usr/lib/jvm/java-8-openjdk-amd64/jre/jre/lib/amd64/server/libjvm.so,
Checking library /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so,
Found VM client definition in configuration,
Checking library /usr/lib/jvm/java-8-openjdk-amd64/jre/jre/lib/amd64/client/libjvm.so,
Checking library /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/client/libjvm.so,
Cannot locate library for VM client (skipping),
Found VM zero definition in configuration,
Checking library /usr/lib/jvm/java-8-openjdk-amd64/jre/jre/lib/amd64/zero/libjvm.so,
Checking library /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/zero/libjvm.so,
Cannot locate library for VM zero (skipping),
Found VM dcevm definition in configuration,
Checking library /usr/lib/jvm/java-8-openjdk-amd64/jre/jre/lib/amd64/dcevm/libjvm.so,
Checking library /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/dcevm/libjvm.so,
Cannot locate library for VM dcevm (skipping),
Java Home located in /usr/lib/jvm/java-8-openjdk-amd64/jre,
+-- DUMPING JAVA HOME STRUCTURE ------------------------,
| Java Home:       "/usr/lib/jvm/java-8-openjdk-amd64/jre",
| Java VM Config.: "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/jvm.cfg",
| Found JVMs:      1,
| JVM Name:        "server",
|                  "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so",
+-------------------------------------------------------,
Running w/ LD_LIBRARY_PATH=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64,
done.,
redirecting stdout to /dev/null and stderr to /dev/null,
Switching umask back to 022 from 077,
Attemtping to load library /lib64/libcap.so.2,
Attemtping to load library /lib64/libcap.so.1,
Attemtping to load library /lib64/libcap.so,
Attemtping to load library /usr/lib64/libcap.so.2,
Attemtping to load library /usr/lib64/libcap.so.1,
Attemtping to load library /usr/lib64/libcap.so,
Attemtping to load library /lib/libcap.so.2,
Attemtping to load library /lib/libcap.so.1,
Attemtping to load library /lib/libcap.so,
Attemtping to load library /usr/lib/libcap.so.2,
Attemtping to load library /usr/lib/libcap.so.1,
Attemtping to load library /usr/lib/libcap.so,
Attemtping to load library libcap.so.2,
loaded cap_free from libcap.,
loaded cap_init from libcap.,
loaded cap_clear from libcap.,
loaded cap_get_flag from libcap.,
loaded cap_set_flag from libcap.,
loaded cap_set_proc from libcap.,
increased capability set.,
user changed to 'unifi-video',
decreased capability set to min required.,
Using default JVM in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so,
Attemtping to load library /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so,
JVM library /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so loaded,
JVM library entry point found (0xBB4CB1E0),
+-- DUMPING JAVA VM CREATION ARGUMENTS -----------------,
| Version:                       0x010004,
| Ignore Unrecognized Arguments: True,
| Extra options:                 17,
|   "-Djava.class.path=/usr/share/java/commons-daemon.jar:/usr/lib/unifi-video/lib/airvision.jar" (0x00000000),
|   "-Dav.tempdir=/var/cache/unifi-video" (0x00000000),
|   "-Djava.security.egd=file:/dev/./urandom" (0x00000000),
|   "-Xms32240M" (0x00000000),
|   "-Xmx32240M" (0x00000000),
|   "-XX:+HeapDumpOnOutOfMemoryError" (0x00000000),
|   "-XX:+UseG1GC" (0x00000000),
|   "-XX:+UseStringDeduplication" (0x00000000),
|   "-Djava.library.path=/usr/lib/unifi-video/lib" (0x00000000),
|   "-Djava.awt.headless=true" (0x00000000),
|   "-Djavax.net.ssl.trustStore=/usr/lib/unifi-video/data/ufv-truststore" (0x00000000),
|   "-Dfile.encoding=UTF-8" (0x00000000),
|   "-Dcom.sun.management.jmxremote" (0x00000000),
|   "-Dcom.sun.management.jmxremote.ssl=false" (0x00000000),
|   "-Dcom.sun.management.jmxremote.authenticate=false" (0x00000000),
|   "-Dcom.sun.management.jmxremote.port=7654" (0x00000000),
|   "-Djava.rmi.server.hostname=172.17.0.6" (0x00000000),
+-------------------------------------------------------,
| Internal options:              4,
|   "-Dcommons.daemon.process.id=117" (0x00000000),
|   "-Dcommons.daemon.process.parent=116" (0x00000000),
|   "-Dcommons.daemon.version=1.0.15-dev" (0x00000000),
|   "abort" (0x9835e4e0),
+-------------------------------------------------------,
Java VM created successfully,
Class org/apache/commons/daemon/support/DaemonLoader found,
Native methods registered,
java_init done,
Daemon loading...,
Waiting for mongodb to come online...2019-02-27 22:53:07,156 ERROR Unable to locate appender ConsoleAppender for logger ,
emon loaded successfully,
java_load done,
dropped capabilities.,

The "Waiting for mongodb to come online." appears to be from this project's bash script.

Below is the command I created the container with

docker run -d \
--name unifi-video \
--restart=always \
--label=com.centurylinklabs.watchtower.enable=true \
--cap-add SYS_ADMIN \
--cap-add DAC_READ_SEARCH \
-p 10001:10001 \
-p 1935:1935 \
-p 6666:6666 \
-p 7080:7080 \
-p 7442:7442 \
-p 7443:7443 \
-p 7444:7444 \
-p 7445:7445 \
-p 7446:7446 \
-p 7447:7447 \
-v /storage/docker/unifi-video/data:/var/lib/unifi-video \
-v /storage/docker/unifi-video/videos:/usr/lib/videos \
-e TZ=America/Chicago \
-e PUID=109 \
-e PGID=115 \
-e DEBUG=1 \
--security-opt apparmor:unconfined \
pducharme/unifi-video-controller

[Caboosse]

Seeing the same issue. Created new dirs for a fresh install to make sure my DB wasnt corrupted. Issues persists on fresh pull.

[NathanTheGr8]

I just noticed this in the read me.

Starting with Unifi Video 3.10.x, in-place upgrades are not fully supported. It's best to generate and save a backup file, and remove your data folder. Then restore the backup from the web setup page.

I will change unifi video to not auto update with watchtower. I will just have to bite the bullet and restore from a backup :(. I don't think I have a recent config backup, but that is my own fault.

[kevsterd]

Reading the Unifi release notes they say

To ensure a smooth upgrade path, NVR appliances must upgrade to 3.9.12 before upgrading to 3.10.0+

Looking at the Docker tags, I don't believe a 3.9.12 tag Docker contaner was ever released. We may need this along with some notes here how to upgrade.....

[squirtbrnr]

I am seeing the same thing. I think our issues are related #128

[NathanTheGr8]

Possible, I ended up recreating my setup instead of fixing the issue. I will plan to do manual updates to this container in the future.

[chelming]

There's a tag for 3.9.12 now (and it works for me).

[chelming]

Funny enough though, the error I'm seeing is

2019-03-04T16:24:26.936-0500 I STORAGE  [initandlisten] exception in initAndListen: 28662 Cannot start server. Detected data files in /usr/lib/unifi-video/data/db created by the 'wiredTiger' storage engine, but the specified storage engine was 'mmapv1'., terminating```

[chelming]

Not looking too good based on UBNT's release notes: https://community.ubnt.com/t5/UniFi-Video-Blog/UniFi-Video-3-10-1-Soft-Release/ba-p/2658279

upgrading an NVR from 3.9.12 to 3.10.0+ must be done via the web UI update. Subsequent updates can then be performed via apt, manual, etc. unless otherwise noted in the release notes.

[chelming]

@fryfrog were you able to upgrade from 3.9. to 3.10 or did you create a new instance?

[fryfrog]

I've been using the same unifi-video instance that I started with years ago, born outside a Docker container. I do all the unifi-video Docker changes in my own repo and make sure mine works before I even merge it to the beta channel.

So yes, mine has been working the whole time you guys have all been having issues and I still have no idea why. :/

Edit: In that time, all of my troubles have stemmed from the featureVersion in mongodb. Instead of pinning the mongodb version to 3.2 or 3.4 or 3.6, I've been updating that in my db and running the latest monodb version. Which is what I switched the container to start doing too.

[fryfrog]

Does anyone know anything about wiredtiger versus mmapv1?

[acantril]

A fix for this ... edit /usr/lib/unifi-video/conf/mongodv3.0+.conf replace engine: mmapv1 with engine: wiredTiger

then restart the container.

[acantril]

Would be good to get this changed in the image.

[fryfrog]

Wouldn't anyone actually using wiredTiger be boned then?

[kevsterd]

I had a bad experiance updating. In the end I have removed the container, unmanaged the cameras and rebuilt a fresh directory and container. No matter what I tried I was getting errors all over the shop, some relating to chrome I suspect as Edge was stable for the config. I didnt have infinite ammount of time to debug.

Anyway CPU on my Synology is v low and its consuming about 2gb of RAM. Happy with that.

[chelming]

fwiw, I created a brand new container, grabbed the latest .zip backup from the backups folder in my Unifi Video volume and loaded that into the new container. Everything is working again. 🤷‍♂️

[fryfrog]

@acantril: Do you know what the heck the mongodv3.0+.conf files are used for?

I don't remember doing anything to make it happen, but my unifi-video db is wiredTiger. :/

[fryfrog]

So 3.10.2 uses a mongod-wt.conf which is really wiredTiger. Older versions don't. I'm closing this, but for future searchers... the overall solution is probably to upgrade the db from mmapv1 to wiredTiger and then put it in the right place, the db-wt folder.