search

roflcoopter / viseron

Self-hosted, local only NVR and AI Computer Vision software. With features such as object detection, motion detection, face recognition and more, it gives you the power to keep an eye on your home, office or any other place you want to monitor.
MIT License
1.5k stars 154 forks source link

Viseron Won't Start With Deepstack #775

Closed tdashmike closed 1 week ago

tdashmike commented 1 week ago

Hello, I moved my containers to a new computer and I can't get Viseron to start with Deepstack. The deepstack website loads at the same port number.

logger:
  default_level: debug

ffmpeg:
  camera:
    frontdoor:  # This value has to be unique across all cameras
      name: T-FRONTDOOR
      host: 192.168.1.4
      port: 8554
      path: "/T-FRONTDOOR_LOW"

      width: 640
      height: 360
      fps: 30
      codec: h264
      rtsp_transport: tcp

      recorder:
        retain: 14

mog2:
  motion_detector:
    cameras:
      frontdoor:
        trigger_recorder: false
        threshold: 15
        area: 0.005
        fps: 5

deepstack:
  host: 192.168.7.4
  port: 5500
  api_key: aaabbb
  object_detector:
#    dnn_backend: openvino
#    dnn_target: opencl
    cameras:
      frontdoor:
        scan_on_motion_only: true
        fps: 1

nvr:
  frontdoor:
[cont-init.d] executing container initialization scripts...
[cont-init.d] 10-adduser: executing... 
usermod: no changes
************************ UID/GID *************************
User uid:    0
User gid:    0
************************** Done **************************
[cont-init.d] 10-adduser: exited 0.
[cont-init.d] 20-gid-video-device: executing... 
[cont-init.d] 20-gid-video-device: exited 0.
[cont-init.d] 30-edgetpu-permission: executing... 
************** Setting EdgeTPU permissions ***************
Coral Vendor IDs:
0=1 1=a 2=6 3=e 
0=1 1=8 2=d 3=1 
No EdgeTPU USB device was found
No EdgeTPU PCI device was found
************************** Done **************************
[cont-init.d] 30-edgetpu-permission: exited 0.
[cont-init.d] 40-set-env-vars: executing... 
****** Checking for hardware acceleration platforms ******
OpenCL cannot be used
VA-API is available!
CUDA cannot be used
*********************** Done *****************************
[cont-init.d] 40-set-env-vars: exited 0.
[cont-init.d] 50-check-if-rpi: executing... 
********** Checking if we are running on an RPi **********
Not running on any supported RPi
*********************** Done *****************************
[cont-init.d] 50-check-if-rpi: exited 0.
[cont-init.d] 55-check-if-jetson: executing... 
****** Checking if we are running on a Jetson Board ******
Not running on any supported Jetson board
*********************** Done *****************************
[cont-init.d] 55-check-if-jetson: exited 0.
[cont-init.d] 60-ffmpeg-path: executing... 
****************** Getting FFmpeg path *******************
FFmpeg path: /home/abc/bin/ffmpeg
*********************** Done *****************************
[cont-init.d] 60-ffmpeg-path: exited 0.
[cont-init.d] 70-gstreamer-path: executing... 
***************** Getting GStreamer path *****************
GStreamer path: /usr/bin/gst-launch-1.0
*********************** Done *****************************
[cont-init.d] 70-gstreamer-path: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2024-06-22 15:39:53] [INFO    ] [viseron.core] - -------------------------------------------
[2024-06-22 15:39:53] [INFO    ] [viseron.core] - Initializing Viseron
[2024-06-22 15:39:53] [INFO    ] [viseron.components] - Setting up component logger
[2024-06-22 15:39:53] [INFO    ] [viseron.components] - Setup of component logger took 0.0 seconds
[2024-06-22 15:39:53] [INFO    ] [viseron.components] - Setting up component data_stream
[2024-06-22 15:39:53] [DEBUG   ] [viseron.components.data_stream] - ulimit -u output: unlimited
[2024-06-22 15:39:53] [DEBUG   ] [viseron.components.data_stream] - Max threads: 999999
[2024-06-22 15:39:53] [DEBUG   ] [viseron.watchdog] - Registering <RestartableThread(data_stream, initial daemon)> in the watchdog
[2024-06-22 15:39:53] [INFO    ] [viseron.components] - Setup of component data_stream took 0.0 seconds
[2024-06-22 15:39:53] [INFO    ] [viseron.components] - Setting up component webserver
[2024-06-22 15:39:53] [DEBUG   ] [asyncio] - Using selector: EpollSelector
[2024-06-22 15:39:53] [DEBUG   ] [viseron.components.data_stream] - Subscribing to data topic event/domain/registered/camera, <bound method Webserver.camera_registered of <Webserver(Tornado Webserver, initial daemon)>>
[2024-06-22 15:39:53] [DEBUG   ] [viseron.components.data_stream] - Subscribing to data topic event/domain/setup/domain_failed/camera/*, <bound method Webserver.camera_registered of <Webserver(Tornado Webserver, initial daemon)>>
[2024-06-22 15:39:53] [DEBUG   ] [viseron.components.data_stream] - Subscribing to data topic viseron/signal/shutdown, <bound method Webserver.stop of <Webserver(Tornado Webserver, initial daemon)>>
[2024-06-22 15:39:53] [INFO    ] [viseron.components] - Setup of component webserver took 0.0 seconds
[2024-06-22 15:39:53] [INFO    ] [viseron.components] - Setting up component deepstack
[2024-06-22 15:39:53] [INFO    ] [viseron.components] - Setting up component ffmpeg
[2024-06-22 15:39:53] [INFO    ] [viseron.components] - Setting up component nvr
[2024-06-22 15:39:53] [INFO    ] [viseron.components] - Setup of component nvr took 0.0 seconds
[2024-06-22 15:39:53] [INFO    ] [viseron.components] - Setup of component ffmpeg took 0.0 seconds
[viseron-finish] Viseron exit code 256
[viseron-finish] Viseron received signal 4
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
roflcoopter commented 6 days ago

Seen this happen with compreface as well, did you manage to fix it?

tdashmike commented 6 days ago

I added deepstack back in and it just worked last night. Let me try putting the config above back in and see what happens. Do you know how I can get more logging? The deepstack container didn't have any entries and the viseron logs didn't seem to show anything with the debug logging level. Thanks!

roflcoopter commented 6 days ago

It looks like a segfault from the logs, so if you check dmesg on your host when it happens you might get a clue

tdashmike commented 6 days ago

OK, so it looks like it's because I was using x86-64-v2-AES as the processor type in Proxmox. I changed it to "host" type earlier in the day so that's why it worked that night. Viseron is running in a docker container on a Ubuntu 24.04 Server VM.

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 10-adduser: executing... 
usermod: no changes
************************ UID/GID *************************
User uid:    0
User gid:    0
************************** Done **************************
[cont-init.d] 10-adduser: exited 0.
[cont-init.d] 20-gid-video-device: executing... 
[cont-init.d] 20-gid-video-device: exited 0.
[cont-init.d] 30-edgetpu-permission: executing... 
************** Setting EdgeTPU permissions ***************
Coral Vendor IDs:
0=1 1=a 2=6 3=e 
0=1 1=8 2=d 3=1 
No EdgeTPU USB device was found
No EdgeTPU PCI device was found
************************** Done **************************
[cont-init.d] 30-edgetpu-permission: exited 0.
[cont-init.d] 40-set-env-vars: executing... 
****** Checking for hardware acceleration platforms ******
OpenCL is available!
VA-API is available!
CUDA cannot be used
PostgreSQL major version: 14
PostgreSQL bin: /usr/lib/postgresql/14/bin
*********************** Done *****************************
[cont-init.d] 40-set-env-vars: exited 0.
[cont-init.d] 50-check-if-rpi: executing... 
********** Checking if we are running on an RPi **********
Not running on any supported RPi
*********************** Done *****************************
[cont-init.d] 50-check-if-rpi: exited 0.
[cont-init.d] 55-check-if-jetson: executing... 
****** Checking if we are running on a Jetson Board ******
Not running on any supported Jetson board
*********************** Done *****************************
[cont-init.d] 55-check-if-jetson: exited 0.
[cont-init.d] 60-ffmpeg-path: executing... 
****************** Getting FFmpeg path *******************
FFmpeg path: /home/abc/bin/ffmpeg
*********************** Done *****************************
[cont-init.d] 60-ffmpeg-path: exited 0.
[cont-init.d] 70-gstreamer-path: executing... 
***************** Getting GStreamer path *****************
GStreamer path: /usr/bin/gst-launch-1.0
*********************** Done *****************************
[cont-init.d] 70-gstreamer-path: exited 0.
[cont-init.d] 80-postgres: executing... 
***************** Preparing PostgreSQL *******************
Database has already been initialized.
*********************** Done *****************************
[cont-init.d] 80-postgres: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
Starting PostgreSQL Server...
/var/run/postgresql:5432 - no response
Waiting for PostgreSQL Server to start...
2024-06-26 23:56:04.850 UTC [468] LOG:  starting PostgreSQL 14.12 (Ubuntu 14.12-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
2024-06-26 23:56:04.850 UTC [468] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2024-06-26 23:56:04.850 UTC [468] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2024-06-26 23:56:04.865 UTC [468] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-06-26 23:56:04.880 UTC [470] LOG:  database system was shut down at 2024-06-26 23:53:57 UTC
2024-06-26 23:56:04.891 UTC [468] LOG:  database system is ready to accept connections
/var/run/postgresql:5432 - accepting connections
PostgreSQL Server has started!
[2024-06-26 23:56:06] [INFO    ] [viseron.core] - -------------------------------------------
[2024-06-26 23:56:06] [INFO    ] [viseron.core] - Initializing Viseron 3.0.0b7
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setting up component logger
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setup of component logger took 0.0 seconds
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setting up component data_stream
[2024-06-26 23:56:06] [DEBUG   ] [viseron.components.data_stream] - ulimit -u output: unlimited
[2024-06-26 23:56:06] [DEBUG   ] [viseron.components.data_stream] - Max threads: 999999
[2024-06-26 23:56:06] [DEBUG   ] [viseron.watchdog] - Registering <RestartableThread(data_stream, initial daemon)> in the watchdog
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setup of component data_stream took 0.0 seconds
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setting up component webserver
[2024-06-26 23:56:06] [DEBUG   ] [asyncio] - Using selector: EpollSelector
[2024-06-26 23:56:06] [DEBUG   ] [viseron.components.data_stream] - Subscribing to data topic viseron/signal/shutdown, <bound method Webserver.stop of <Webserver(Tornado Webserver, initial daemon)>>
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setup of component webserver took 0.0 seconds
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setting up component storage
[2024-06-26 23:56:06] [INFO    ] [alembic.runtime.migration] - Context impl PostgresqlImpl.
[2024-06-26 23:56:06] [INFO    ] [alembic.runtime.migration] - Will assume transactional DDL.
[2024-06-26 23:56:06] [DEBUG   ] [viseron.components.storage] - Current database revision: 8462ca6851b2
[2024-06-26 23:56:06] [DEBUG   ] [viseron.components.data_stream] - Subscribing to data topic event/domain/registered/camera, <bound method Storage._camera_registered of <viseron.components.storage.Storage object at 0x7fb058c28640>>
[2024-06-26 23:56:06] [DEBUG   ] [viseron.components.data_stream] - Subscribing to data topic viseron/signal/stopping, <bound method Storage._shutdown of <viseron.components.storage.Storage object at 0x7fb058c28640>>
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setup of component storage took 0.0 seconds
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setting up component nvr
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setup of component nvr took 0.0 seconds
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setting up component ffmpeg
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setting up component mog2
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setting up component deepstack
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setup of component ffmpeg took 0.0 seconds
[2024-06-26 23:56:06] [INFO    ] [viseron.components] - Setup of component mog2 took 0.0 seconds
[viseron-finish] Viseron exit code 256
[viseron-finish] Viseron received signal 4
[cont-finish.d] executing container finish scripts...
[cont-finish.d] 10-postgres: executing... 
Wait for Viseron to stop...
error: list of process IDs must follow -p

Usage:
 ps [options]

 Try 'ps --help <simple|list|output|threads|misc|all>'
  or 'ps --help <s|l|o|t|m|a>'
 for additional help text.

For more details see ps(1).
Viseron has stopped!
Stopping PostgreSQL...
2024-06-26 23:56:06.709 UTC [468] LOG:  received fast shutdown request
2024-06-26 23:56:06.716 UTC [468] LOG:  aborting any active transactions
2024-06-26 23:56:06.718 UTC [468] LOG:  background worker "logical replication launcher" (PID 476) exited with exit code 1
2024-06-26 23:56:06.719 UTC [471] LOG:  shutting down
2024-06-26 23:56:06.785 UTC [468] LOG:  database system is shut down
waiting for server to shut down.... done
server stopped
Waiting for PostgreSQL Server to stop...
/var/run/postgresql:5432 - no response
PostgreSQL Server has stopped!
[cont-finish.d] 10-postgres: exited 0.
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.

Error:

 traps: python3[7189] trap invalid opcode ip:7fb01e78f740 sp:7fb037df8860 error:0 in _dlib_pybind11.cpython-310-x86_64-linux-gnu.so[7fb01e689000+781000]
tdashmike commented 4 days ago

Viseron will run now but I suspect it is causing my host to freeze. I had Viseron turned off all day and the host has not crashed yet. As soon as I turn it on I get these 2 errors. Host is still running but I'll see it it lasts the night.

[79747.940175] python3[1726015]: segfault at 0 ip 0000786985d8b75d sp 00007ffe26edcb08 error 4 in libc.so.6[786985c28000+188000] likely on CPU 1 (core 1, socket 0)
[79747.940186] Code: 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 89 f8 48 89 fa c5 f9 ef c0 25 ff 0f 00 00 3d e0 0f 00 00 0f 87 33 01 00 00 <c5> fd 74 0f c5 fd d7 c1 85 c0 74 57 f3 0f bc c0 c5 f8 77 c3 66 66
[79748.228355] python3[1726033]: segfault at 0 ip 000078b90118b75d sp 00007fff174fc088 error 4 in libc.so.6[78b901028000+188000] likely on CPU 3 (core 3, socket 0)
[79748.228366] Code: 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 89 f8 48 89 fa c5 f9 ef c0 25 ff 0f 00 00 3d e0 0f 00 00 0f 87 33 01 00 00 <c5> fd 74 0f c5 fd d7 c1 85 c0 74 57 f3 0f bc c0 c5 f8 77 c3 66 66
tdashmike commented 4 days ago

Logs from the crash this morning. The dmesg log is from the VM running the Viseron container. viseron_log.txt VM dmesg.txt