search

jakowenko / double-take

Unified UI and API for processing and training images for facial recognition.
https://hub.docker.com/r/jakowenko/double-take
MIT License
1.22k stars 97 forks source link

[BUG] Won't connect to DeepStack #167

Open TheGroundZero opened 2 years ago

TheGroundZero commented 2 years ago

SOLVED Solution provided by @szmidtpiotr works! :) https://github.com/jakowenko/double-take/issues/167#issuecomment-975376875

Describe the bug I have installed DeepStack as a seperate Docker container. Forwarded port 5000 to 4999 (on the host). This used to be port 80, but I thought some conflict could've been the issue, which it wasn't. I am using a key. Double Take won't connect to my DeepStack. Frigate works fine.

Version of Double Take v1.6.0-893328f

Expected behavior Double Take should connect to my DeepStack detector If there are any errors, they should be shown in the logs.

Screenshots image image image

Hardware

Additional context

# Double Take
mqtt:
  host: 192.168.10.105
  user: <USER>
  password: <PASS>

frigate:
  url: http://192.168.10.105:5000

detectors:
  deepstack:
    url: http://192.168.10.105:4999
    key: <KEY>
docker run -d --restart=unless-stopped --name="deepstack" -e VISION-DETECTION=True -e VISION-FACE=True -e API-KEY="<MY KEY>" -v localstorage:/datastore -p 80:5000 deepquestai/deepstack

Tested both with and without key.

gianlazz commented 2 years ago

I'm having the same issue.

themusicnerd commented 2 years ago

Exact same issue here. Running in seperate dockers on the latest unraid.

I even tried moving my frigate port to 5001 leaving deepstack at 5000. Still no luck.

ozett commented 2 years ago

i have this working, but connection to deepstack is unreliable.. image maybe you comment out your "key" in config? - or its a bug?

TheGroundZero commented 2 years ago

i have this working, but connection to deepstack is unreliable.. image maybe you comment out your "key" in config? - or its a bug?

I tried both with and without key (restarting the container with and without key env var of course). Doesn't help.

TheGroundZero commented 2 years ago

Should I try with the DeepStack add-on from the DoubleTake add-ons repo? But then if my Coral ever arrives I'd need to switch add-ons?

ozett commented 2 years ago

i started it like this: sudo docker run --gpus all -e VISION-FACE=True -v localstorage:/datastore -p 5001:5000 deepquestai/deepstack:gpu its running on the host perfectly since start. dont know when it quits... double-take looses sometimes connection, or deepstack quits working. dont know. but the container stays running ... sometimes i start it all again... (maybe nvidia gpu issue?). its all running on ubuntu 20.04 host..

TheGroundZero commented 2 years ago

I launch my container like this:

docker run -d --restart=unless-stopped --name="deepstack" -e VISION-DETECTION=True -e VISION-FACE=True -e API-KEY="<MY KEY>" -v localstorage:/datastore -p 80:5000 deepquestai/deepstack

It's accessible via my browser on http://<IP.of.Home.Assistant> (cfr. screenshot in OP) but Double Take won't connect to it.

ozett commented 2 years ago

my try different port-setting in docker command? -p 5001:5000 wich is: -p use-this-nr-alsoin-deepstack-config-:5000

TheGroundZero commented 2 years ago

my try different port-setting in docker command? -p 5001:5000 wich is: -p use-this-nr-alsoin-deepstack-config-:5000

Tried 4999:5000 already 5001 was already taken, by Frigate iirc

szmidtpiotr commented 2 years ago

https://www.reddit.com/r/homeassistant/comments/qi53h5/comment/hjz4ci7/?utm_source=share&utm_medium=web2x&context=3

Bug is related to Virtualiization of the CPU as i understand. If you use proxmox to setup HASSIO then simply change CPU type to HOST in your HA vm. Helped for me

jdguillot commented 2 years ago

I'm having this same problem but I am not running it within Home Assistant. I followed the guide to get Frigate running in a Turnkey Core based Proxmox LXC container. I am then running Double Take and Deepstack as Docker containers on that same LXC Container. The LXC container is privileged and nesting is enabled.

I also tried running Deepstack on a seperate Ubuntu based VM that did have Host CPU enabled but it still wouldn't connect. The odd thing there was I could see images being processed in the deepstack log but Double Take never turned to a green dot.

Running latest Double Take (1.6.0-893328f) and Deepstack v2021.06.1. Frigate connects just fine in Double Take.

ozett commented 2 years ago

my try different port-setting in docker command? -p 5001:5000 wich is: -p use-this-nr-alsoin-deepstack-config-:5000

Tried 4999:5000 already 5001 was already taken, by Frigate iirc

i dont have unraid, but may you check port-settings on unraid. ? https://wiki.unraid.net/UnRAID_6_2/Docker_Management

TheGroundZero commented 2 years ago

my try different port-setting in docker command? -p 5001:5000 wich is: -p use-this-nr-alsoin-deepstack-config-:5000

Tried 4999:5000 already 5001 was already taken, by Frigate iirc

i dont have unraid, but may you check port-settings on unraid. ? https://wiki.unraid.net/UnRAID_6_2/Docker_Management

As mentioned in OP, I don't run on Unraid either, I use Proxmox. I'll try @szmidtpiotr's linked solution

TheGroundZero commented 2 years ago

@szmidtpiotr Is it advisable to enable hv_evmcs? image

szmidtpiotr commented 2 years ago

cpu

To be honest i dont know. Im not an expert, just found that sollution in other place, that worked for me - but if you could test and give us update that would be grate.

TheGroundZero commented 2 years ago

The solution definitly worked :)

image I did need to restart my Proxmox system after adding the config as I was unable to reload the kvm_intel module.

  1. Check if nested virtualization is enabled on the Proxmox server

    cat /sys/module/kvm_intel/parameters/nested
N

  2. Activate the parameter

    echo "options kvm-intel nested=Y" > /etc/modprobe.d/kvm-intel.conf

  3. Activate virtualization on guest

    qm set 104 --cpu host # Update VM ID to match yours

    Or via Web Ui: image

  4. Reboot Host

    cat /sys/module/kvm_intel/parameters/nested
# Y

Enabling hv_evmcs caused an error in my setup. As I don't fully understand the effect of this option, I won't fiddle with it too much ;)

image

TheGroundZero commented 2 years ago

Next up: fix these two issues:

  1. Double Take doesn't find the clips from Frigate image

  2. Frigate can't connect to its own api/stats image

szmidtpiotr commented 2 years ago

Use ip address of the frigate server not localhost generic name used by HA as i assume. If you use theFrigate addon to HA give ip addres of HA server with :5000 port in your Configuration -> Integrations -> Frigate plugin

TheGroundZero commented 2 years ago

Use ip address of the frigate server not localhost generic name used by HA as i assume. If you use theFrigate addon to HA give ip addres of HA server with :5000 port in your Configuration -> Integrations -> Frigate plugin

Deleted the Frigate integration and re-added it using http://192.168.10.105:5000 as address instead of http://ccab4aaf-frigate-beta:5000 (which is mentioned in the docs)

Double Take not getting snapshots also seems to have solved itself.

gianlazz commented 2 years ago

https://www.reddit.com/r/homeassistant/comments/qi53h5/comment/hjz4ci7/?utm_source=share&utm_medium=web2x&context=3

Bug is related to Virtualiization of the CPU as i understand. If you use proxmox to setup HASSIO then simply change CPU type to HOST in your HA vm. Helped for me

Yup, this was apparently my issue too. I'm running via proxmox and switching home assistant's cpu type to host allowed deepstack to be connected to.

ericfri commented 2 years ago

I too am having this issue. I'm running everything in Unraid. I've tried a few different versions of Deepstack and it won't connect, although I can see the requests being submitted, but the dot stays red.

ryannamani commented 2 years ago

Same Issue

image

Logs

image

I can see my deepstack docker is up and running (refernce below)

image

I have also checked regarding the Vitualization as i'm using Intel Nuc and it looks good as well

image

Am i missing something ?

jakowenko commented 2 years ago

@rakeshyannamani what do your deepstack container logs show?

ryannamani commented 2 years ago

@rakeshyannamani what do your deepstack container logs show?

this is what I see under Deepstack Docker Logs

image
LordNex commented 2 years ago

I'm having this same issue. Just started today. I haven't made any changes or updates. I just rebooted my Jetson Nano through SSH and when it comes up, Double Take connects to it, but after a few min goes red. Frigate is still working. Everything is on seprate boxes shown below. Any ideas?

Deepstack: Jetson Nano 4gig Frigate: RPi4 8gig Home Assistant and Double Take: i5 Laptop with 8 gigs of RAM

LordNex commented 2 years ago

I'm not running on a Proxmox system or anything. I have DeepStack installed on a Jetson Nano 4gig, Frigate installed on a RPi 4 8gig, and Home Assistant with Double Take setup on VMWare ESXi running on an i5 laptop with 8gigs. Here is a screenshot of everything. Anyone have any ideas? I can sometimes get it to connect aftter a reboot for a few min then it goes red again.

deepstackerror .

LordNex commented 2 years ago

HOnestly it seems like DeepStack is still working. Maybe its just the way Double Take checks to see of DeepStack is alive maybe?


21-12-18 17:31:43 info: processing livingroom: 16ab9000-ed16-4c5a-bfc0-a08d2b4e7d94
21-12-18 17:31:56 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:31:56 warn: sleeping for 0.5 second(s)
21-12-18 17:31:56 info: done processing livingroom: 16ab9000-ed16-4c5a-bfc0-a08d2b4e7d94 in 15.52 sec
21-12-18 17:31:56 info: {
  id: '16ab9000-ed16-4c5a-bfc0-a08d2b4e7d94',
  duration: 15.52,
  timestamp: '2021-12-18T23:31:56.516Z',
  attempts: 1,
  camera: 'livingroom',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 17:31:56 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:31:56 warn: sleeping for 0.5 second(s)
21-12-18 17:31:57 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:31:57 warn: sleeping for 0.5 second(s)
21-12-18 17:32:12 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:32:12 warn: sleeping for 1 second(s)
21-12-18 17:32:13 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:32:13 warn: sleeping for 1 second(s)
21-12-18 17:32:29 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:32:29 warn: sleeping for 1.5 second(s)
21-12-18 17:32:46 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:32:46 warn: sleeping for 2 second(s)
21-12-18 17:32:47 info: processing livingroom: 05f95ed7-f81e-4f31-b114-f0c679b6dfaa
21-12-18 17:33:02 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:33:02 warn: sleeping for 2.5 second(s)
21-12-18 17:33:02 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:33:02 warn: sleeping for 0.5 second(s)
21-12-18 17:33:02 info: done processing livingroom: 05f95ed7-f81e-4f31-b114-f0c679b6dfaa in 15.51 sec
21-12-18 17:33:02 info: {
  id: '05f95ed7-f81e-4f31-b114-f0c679b6dfaa',
  duration: 15.51,
  timestamp: '2021-12-18T23:33:02.903Z',
  attempts: 1,
  camera: 'livingroom',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 17:33:21 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:33:21 warn: sleeping for 3 second(s)
21-12-18 17:33:39 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:33:39 warn: sleeping for 3.5 second(s)
21-12-18 17:33:57 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:33:57 warn: sleeping for 4 second(s)
21-12-18 17:34:16 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:34:16 warn: sleeping for 4.5 second(s)
21-12-18 17:34:38 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 17:34:38 warn: sleeping for 5 second(s)
21-12-18 17:34:43 info: done processing livingroom: 1639870275.944793-kgfn2l in 113.89 sec
21-12-18 17:34:43 info: {
  id: '1639870275.944793-kgfn2l',
  duration: 113.89,
  timestamp: '2021-12-18T23:34:43.099Z',
  attempts: 12,
  camera: 'livingroom',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 18:10:03 info: processing livingroom: 1639872597.904841-awo44n
21-12-18 18:10:03 info: processing livingroom: c0a0d7d7-abb4-48f5-a917-9248734e837f
21-12-18 18:10:18 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:10:18 warn: sleeping for 0.5 second(s)
21-12-18 18:10:19 info: done processing livingroom: c0a0d7d7-abb4-48f5-a917-9248734e837f in 15.52 sec
21-12-18 18:10:19 info: {
  id: 'c0a0d7d7-abb4-48f5-a917-9248734e837f',
  duration: 15.52,
  timestamp: '2021-12-19T00:10:19.431Z',
  attempts: 1,
  camera: 'livingroom',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 18:10:19 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:10:19 warn: sleeping for 0.5 second(s)
21-12-18 18:10:19 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:10:19 warn: sleeping for 0.5 second(s)
21-12-18 18:10:35 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:10:35 warn: sleeping for 1 second(s)
21-12-18 18:10:35 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:10:35 warn: sleeping for 1 second(s)
21-12-18 18:10:51 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:10:51 warn: sleeping for 1.5 second(s)
21-12-18 18:11:08 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:11:08 warn: sleeping for 2 second(s)
21-12-18 18:11:29 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:11:29 warn: sleeping for 2.5 second(s)
21-12-18 18:11:47 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:11:47 warn: sleeping for 3 second(s)
21-12-18 18:12:05 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:12:05 warn: sleeping for 3.5 second(s)
21-12-18 18:12:24 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:12:24 warn: sleeping for 4 second(s)
21-12-18 18:12:47 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:12:47 warn: sleeping for 4.5 second(s)
21-12-18 18:13:06 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:13:06 warn: sleeping for 5 second(s)
21-12-18 18:13:11 info: done processing livingroom: 1639872597.904841-awo44n in 181.06 sec
21-12-18 18:13:11 info: {
  id: '1639872597.904841-awo44n',
  duration: 181.06,
  timestamp: '2021-12-19T00:13:11.814Z',
  attempts: 12,
  camera: 'livingroom',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 18:33:01 info: processing hallcam: 1639873970.310104-fawfn8
21-12-18 18:33:01 info: processing hallcam: 5125a0b9-e4aa-41da-b66c-3603f5fc758b
21-12-18 18:33:13 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:33:13 warn: sleeping for 0.5 second(s)
21-12-18 18:33:14 info: done processing hallcam: 5125a0b9-e4aa-41da-b66c-3603f5fc758b in 15.52 sec
21-12-18 18:33:14 info: {
  id: '5125a0b9-e4aa-41da-b66c-3603f5fc758b',
  duration: 15.52,
  timestamp: '2021-12-19T00:33:14.086Z',
  attempts: 1,
  camera: 'hallcam',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 18:33:14 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:33:14 warn: sleeping for 0.5 second(s)
21-12-18 18:33:14 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:33:14 warn: sleeping for 0.5 second(s)
21-12-18 18:33:29 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:33:29 warn: sleeping for 1 second(s)
21-12-18 18:33:30 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:33:30 warn: sleeping for 1 second(s)
21-12-18 18:33:49 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:33:49 warn: sleeping for 1.5 second(s)
21-12-18 18:34:06 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:34:06 warn: sleeping for 2 second(s)
21-12-18 18:34:21 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:34:21 warn: sleeping for 2.5 second(s)
21-12-18 18:34:38 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 18:34:38 warn: sleeping for 3 second(s)
21-12-18 18:34:46 error: Error: stream error: timeout of 5000ms exceeded
    at createError (/double-take/api/node_modules/axios/lib/core/createError.js:16:15)
    at RedirectableRequest.handleRequestTimeout (/double-take/api/node_modules/axios/lib/adapters/http.js:328:16)
    at RedirectableRequest.emit (node:events:390:28)
    at Timeout.<anonymous> (/double-take/api/node_modules/follow-redirects/index.js:164:12)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7)
21-12-18 18:34:51 error: Error: stream error: timeout of 5000ms exceeded
    at createError (/double-take/api/node_modules/axios/lib/core/createError.js:16:15)
    at RedirectableRequest.handleRequestTimeout (/double-take/api/node_modules/axios/lib/adapters/http.js:328:16)
    at RedirectableRequest.emit (node:events:390:28)
    at Timeout.<anonymous> (/double-take/api/node_modules/follow-redirects/index.js:164:12)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7)
21-12-18 18:34:56 error: Error: stream error: timeout of 5000ms exceeded
    at createError (/double-take/api/node_modules/axios/lib/core/createError.js:16:15)
    at RedirectableRequest.handleRequestTimeout (/double-take/api/node_modules/axios/lib/adapters/http.js:328:16)
    at RedirectableRequest.emit (node:events:390:28)
    at Timeout.<anonymous> (/double-take/api/node_modules/follow-redirects/index.js:164:12)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7)
21-12-18 18:35:01 error: Error: stream error: timeout of 5000ms exceeded
    at createError (/double-take/api/node_modules/axios/lib/core/createError.js:16:15)
    at RedirectableRequest.handleRequestTimeout (/double-take/api/node_modules/axios/lib/adapters/http.js:328:16)
    at RedirectableRequest.emit (node:events:390:28)
    at Timeout.<anonymous> (/double-take/api/node_modules/follow-redirects/index.js:164:12)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7)
21-12-18 18:35:01 info: done processing hallcam: 1639873970.310104-fawfn8 in 122.86 sec
21-12-18 18:35:01 info: {
  id: '1639873970.310104-fawfn8',
  duration: 122.86,
  timestamp: '2021-12-19T00:35:01.898Z',
  attempts: 8,
  camera: 'hallcam',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 19:36:24 info: processing hallcam: 1639877777.65716-lknwh8
21-12-18 19:36:24 info: processing hallcam: 38d55b3b-6776-45f0-9e9f-8d79f575f6ca
21-12-18 19:36:42 info: processing hallcam: 05a7c466-2c87-4f51-b705-0f4209f10c7f
21-12-18 19:36:43 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:36:43 warn: sleeping for 0.5 second(s)
21-12-18 19:36:43 info: done processing hallcam: 38d55b3b-6776-45f0-9e9f-8d79f575f6ca in 1.24 sec
21-12-18 19:36:43 info: {
  id: '38d55b3b-6776-45f0-9e9f-8d79f575f6ca',
  duration: 1.24,
  timestamp: '2021-12-19T01:36:43.997Z',
  attempts: 1,
  camera: 'hallcam',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 19:36:44 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:36:44 warn: sleeping for 0.5 second(s)
21-12-18 19:36:44 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:36:44 warn: sleeping for 0.5 second(s)
21-12-18 19:36:57 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:36:57 warn: sleeping for 0.5 second(s)
21-12-18 19:36:58 info: done processing hallcam: 05a7c466-2c87-4f51-b705-0f4209f10c7f in 15.52 sec
21-12-18 19:36:58 info: {
  id: '05a7c466-2c87-4f51-b705-0f4209f10c7f',
  duration: 15.52,
  timestamp: '2021-12-19T01:36:58.274Z',
  attempts: 1,
  camera: 'hallcam',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 19:36:59 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:36:59 warn: sleeping for 1 second(s)
21-12-18 19:36:59 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:36:59 warn: sleeping for 1 second(s)
21-12-18 19:37:19 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:37:19 warn: sleeping for 1.5 second(s)
21-12-18 19:37:36 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:37:36 warn: sleeping for 2 second(s)
21-12-18 19:37:57 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:37:57 warn: sleeping for 2.5 second(s)
21-12-18 19:38:15 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:38:15 warn: sleeping for 3 second(s)
21-12-18 19:38:37 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:38:37 warn: sleeping for 3.5 second(s)
21-12-18 19:38:56 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:38:56 warn: sleeping for 4 second(s)
21-12-18 19:39:18 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:39:18 warn: sleeping for 4.5 second(s)
21-12-18 19:39:38 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:39:38 warn: sleeping for 5 second(s)
21-12-18 19:39:47 info: done processing hallcam: 1639877777.65716-lknwh8 in 166.32 sec
21-12-18 19:39:47 info: {
  id: '1639877777.65716-lknwh8',
  duration: 166.32,
  timestamp: '2021-12-19T01:39:47.341Z',
  attempts: 12,
  camera: 'hallcam',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 19:41:54 info: processing hallcam: 1639878108.472314-41xblo
21-12-18 19:41:54 info: processing hallcam: 1185e1ab-c90c-4351-8abb-fee2cb134972
21-12-18 19:42:12 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:42:12 warn: sleeping for 0.5 second(s)
21-12-18 19:42:13 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:42:13 warn: sleeping for 0.5 second(s)
21-12-18 19:42:13 info: done processing hallcam: 1185e1ab-c90c-4351-8abb-fee2cb134972 in 15.52 sec
21-12-18 19:42:13 info: {
  id: '1185e1ab-c90c-4351-8abb-fee2cb134972',
  duration: 15.52,
  timestamp: '2021-12-19T01:42:13.475Z',
  attempts: 1,
  camera: 'hallcam',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 19:42:13 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:42:13 warn: sleeping for 0.5 second(s)
21-12-18 19:42:21 info: processing hallcam: 1f64d52e-14d2-41c7-baa9-d8d03180cc58
21-12-18 19:42:28 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:42:28 warn: sleeping for 1 second(s)
21-12-18 19:42:29 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:42:29 warn: sleeping for 1 second(s)
21-12-18 19:42:36 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:42:36 warn: sleeping for 0.5 second(s)
21-12-18 19:42:36 info: done processing hallcam: 1f64d52e-14d2-41c7-baa9-d8d03180cc58 in 15.51 sec
21-12-18 19:42:36 info: {
  id: '1f64d52e-14d2-41c7-baa9-d8d03180cc58',
  duration: 15.51,
  timestamp: '2021-12-19T01:42:36.740Z',
  attempts: 1,
  camera: 'hallcam',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 19:42:48 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:42:48 warn: sleeping for 1.5 second(s)
21-12-18 19:42:49 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:42:49 warn: sleeping for 1.5 second(s)
21-12-18 19:43:06 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:43:06 warn: sleeping for 2 second(s)
21-12-18 19:43:26 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:43:26 warn: sleeping for 2.5 second(s)
21-12-18 19:43:44 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:43:44 warn: sleeping for 3 second(s)
21-12-18 19:44:01 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:44:01 warn: sleeping for 3.5 second(s)
21-12-18 19:44:20 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:44:20 warn: sleeping for 4 second(s)
21-12-18 19:44:42 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:44:42 warn: sleeping for 4.5 second(s)
21-12-18 19:45:02 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 19:45:02 warn: sleeping for 5 second(s)
21-12-18 19:45:07 info: done processing hallcam: 1639878108.472314-41xblo in 157.48 sec
21-12-18 19:45:07 info: {
  id: '1639878108.472314-41xblo',
  duration: 157.48,
  timestamp: '2021-12-19T01:45:07.582Z',
  attempts: 13,
  camera: 'hallcam',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 20:32:26 info: processing livingroom: 1639881140.589116-1x82dn
21-12-18 20:32:26 info: processing livingroom: 6ee3655a-2cbe-43d3-a2a8-64b31e76428d
21-12-18 20:32:44 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 20:32:44 warn: sleeping for 0.5 second(s)
21-12-18 20:32:44 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 20:32:44 warn: sleeping for 0.5 second(s)
21-12-18 20:32:45 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 20:32:45 warn: sleeping for 0.5 second(s)
21-12-18 20:32:45 info: done processing livingroom: 6ee3655a-2cbe-43d3-a2a8-64b31e76428d in 15.52 sec
21-12-18 20:32:45 info: {
  id: '6ee3655a-2cbe-43d3-a2a8-64b31e76428d',
  duration: 15.52,
  timestamp: '2021-12-19T02:32:45.285Z',
  attempts: 1,
  camera: 'livingroom',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 20:33:01 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 20:33:01 warn: sleeping for 1 second(s)
21-12-18 20:33:17 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 20:33:17 warn: sleeping for 1.5 second(s)
21-12-18 20:33:34 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 20:33:34 warn: sleeping for 2 second(s)
21-12-18 20:33:55 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 20:33:55 warn: sleeping for 2.5 second(s)
21-12-18 20:34:14 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 20:34:14 warn: sleeping for 3 second(s)
21-12-18 20:34:33 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 20:34:33 warn: sleeping for 3.5 second(s)
21-12-18 20:34:55 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 20:34:55 warn: sleeping for 4 second(s)
21-12-18 20:35:14 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 20:35:14 warn: sleeping for 4.5 second(s)
21-12-18 20:35:36 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 20:35:36 warn: sleeping for 5 second(s)
21-12-18 20:35:41 info: done processing livingroom: 1639881140.589116-1x82dn in 181.17 sec
21-12-18 20:35:41 info: {
  id: '1639881140.589116-1x82dn',
  duration: 181.17,
  timestamp: '2021-12-19T02:35:41.672Z',
  attempts: 11,
  camera: 'livingroom',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 21:36:04 info: processing hallcam: 1639884960.211439-t3ariw
21-12-18 21:36:04 info: processing hallcam: a652f611-66cf-40a3-be88-45422686323f
21-12-18 21:36:19 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 21:36:19 warn: sleeping for 0.5 second(s)
21-12-18 21:36:19 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 21:36:19 warn: sleeping for 0.5 second(s)
21-12-18 21:36:20 info: done processing hallcam: a652f611-66cf-40a3-be88-45422686323f in 15.52 sec
21-12-18 21:36:20 info: {
  id: 'a652f611-66cf-40a3-be88-45422686323f',
  duration: 15.52,
  timestamp: '2021-12-19T03:36:20.021Z',
  attempts: 1,
  camera: 'hallcam',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 21:36:20 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 21:36:20 warn: sleeping for 0.5 second(s)
21-12-18 21:36:32 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 21:36:32 warn: sleeping for 1 second(s)
21-12-18 21:36:33 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 21:36:33 warn: sleeping for 1 second(s)
21-12-18 21:36:49 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 21:36:49 warn: sleeping for 1.5 second(s)
21-12-18 21:36:53 info: processing hallcam: 24df806d-e984-4cc4-9050-498f6481d5f6
21-12-18 21:37:05 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 21:37:05 warn: sleeping for 2 second(s)
21-12-18 21:37:06 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 21:37:06 warn: sleeping for 0.5 second(s)
21-12-18 21:37:07 info: done processing hallcam: 24df806d-e984-4cc4-9050-498f6481d5f6 in 15.52 sec
21-12-18 21:37:07 info: {
  id: '24df806d-e984-4cc4-9050-498f6481d5f6',
  duration: 15.52,
  timestamp: '2021-12-19T03:37:07.293Z',
  attempts: 1,
  camera: 'hallcam',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 21:37:24 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 21:37:24 warn: sleeping for 2.5 second(s)
21-12-18 21:37:39 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 21:37:39 warn: sleeping for 3 second(s)
21-12-18 21:37:57 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 21:37:57 warn: sleeping for 3.5 second(s)
21-12-18 21:38:01 info: done processing hallcam: 1639884960.211439-t3ariw in 72.7 sec
21-12-18 21:38:01 info: {
  id: '1639884960.211439-t3ariw',
  duration: 72.7,
  timestamp: '2021-12-19T03:38:01.507Z',
  attempts: 9,
  camera: 'hallcam',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 22:32:59 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:32:59 warn: sleeping for 0.5 second(s)
21-12-18 22:33:54 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:33:54 warn: sleeping for 0.5 second(s)
21-12-18 22:42:17 info: Double Take v1.7.0-bf3e911
21-12-18 22:42:19 info: MQTT: connected
21-12-18 22:42:19 info: MQTT: subscribed to frigate/events, frigate/+/person/snapshot
21-12-18 22:42:52 info: Double Take v1.7.0-bf3e911
21-12-18 22:42:53 info: MQTT: connected
21-12-18 22:42:53 info: MQTT: subscribed to frigate/events, frigate/+/person/snapshot
21-12-18 22:43:00 info: purged 42 file(s)
21-12-18 22:55:13 info: processing hallcam: 1639889703.132465-kurwwh
21-12-18 22:55:13 info: processing hallcam: 0bc9a51a-4a74-4189-a310-d7c64ba06da2
21-12-18 22:55:29 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:55:29 warn: sleeping for 0.5 second(s)
21-12-18 22:55:29 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:55:29 warn: sleeping for 0.5 second(s)
21-12-18 22:55:29 info: done processing hallcam: 0bc9a51a-4a74-4189-a310-d7c64ba06da2 in 15.54 sec
21-12-18 22:55:29 info: {
  id: '0bc9a51a-4a74-4189-a310-d7c64ba06da2',
  duration: 15.54,
  timestamp: '2021-12-19T04:55:29.537Z',
  attempts: 1,
  camera: 'hallcam',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 22:55:30 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:55:30 warn: sleeping for 0.5 second(s)
21-12-18 22:55:45 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:55:45 warn: sleeping for 1 second(s)
21-12-18 22:55:45 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:55:45 warn: sleeping for 1 second(s)
21-12-18 22:56:01 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:56:01 warn: sleeping for 1.5 second(s)
21-12-18 22:56:18 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:56:18 warn: sleeping for 2 second(s)
21-12-18 22:56:36 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:56:36 warn: sleeping for 2.5 second(s)
21-12-18 22:56:53 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:56:53 warn: sleeping for 3 second(s)
21-12-18 22:57:12 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:57:12 warn: sleeping for 3.5 second(s)
21-12-18 22:57:45 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:57:45 warn: sleeping for 4 second(s)
21-12-18 22:58:05 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:58:05 warn: sleeping for 4.5 second(s)
21-12-18 22:58:31 error: Error: deepstack process error: timeout of 15000ms exceeded
21-12-18 22:58:31 warn: sleeping for 5 second(s)
21-12-18 22:58:36 info: done processing hallcam: 1639889703.132465-kurwwh in 180.99 sec
21-12-18 22:58:36 info: {
  id: '1639889703.132465-kurwwh',
  duration: 180.99,
  timestamp: '2021-12-19T04:58:36.307Z',
  attempts: 12,
  camera: 'hallcam',
  zones: [],
  matches: [],
  misses: []
}
21-12-18 23:01:00 info: Double Take v1.7.0-bf3e911
21-12-18 23:01:01 info: MQTT: connected
21-12-18 23:01:01 info: MQTT: subscribed to frigate/events, frigate/+/person/snapshot
21-12-18 23:05:44 info: Double Take v1.7.0-bf3e911
21-12-18 23:05:45 info: MQTT: connected
21-12-18 23:05:45 info: MQTT: subscribed to frigate/events, frigate/+/person/snapshot
21-12-18 23:07:06 info: Double Take v1.7.0-bf3e911
21-12-18 23:07:07 info: MQTT: connected
21-12-18 23:07:07 info: MQTT: subscribed to frigate/events, frigate/+/person/snapshot
21-12-18 23:32:41 info: benjamin: queuing 1 file(s) for training
21-12-18 23:35:08 info: benjamin: training complete in 133.04 sec```
LordNex commented 2 years ago

Well I installed your DeepStack (Non-CPU Version) Home Assistant add-on on my VMWare ESXi 7'd Laptop running HA, and got it up and running with Double Take and Frigate. Then left them overnight monitoring a camera that would get a few hits. It did so normally albeit a little slower. But it still shows green and connected. I'm pretty sure this isn't an issue with DeepStack but more of the way DT checks DS for connectivity.

Are there any logs or command results you need to help resolve this? Again thanks so much for such a cool ass application. The way this pulls various formats together with the ease of use it does is outstanding!

LordNex commented 2 years ago

I tried the Beta Addon of Double Take and can confirm that it also shows disconnected from DeepStack running on a Jetson Nano. Works fine if I force it to run on my HA Box but it eats too much processing power. DeepStack and the Jetson work perfectly and are returning results. But for some reason, DT sees it as red.

jakowenko commented 2 years ago

@LordNex in your above logs it looks like the DeepStack API endpoint is timing out. Have you been able to get DeepStack on your Jetson to return a successful response? They have some example ones in their docs if you need reference.

I'm guessing it's showing red on DT because DeepStack is taking too long to respond.

LordNex commented 2 years ago

@jakowenko Yes, it will work for a little while. For instance, I'll reboot the Jetson and verify the docker container is up and accepting connections on its endpoint. I'll then load up double take and start to train images, something that takes a while even on a high processor machine, the Jetson does in half the time. After a few training sessions, I notice connection timed out responses while it's training. Then afterward I can no longer connect to DeepStack unless I reboot both DT and DS. Although I have verified the Jetson and Docker container are both still running and accessible and that the DeepStack container is still able to process requests( I.E. through the Home Assistant service or curl). Yet I can load up the Non-CPU version of DeepStack that comes with your HA add-on and run on it for days and days without a connection problem. Yet it's very slow and not very responsive for my use case. Thus the question about using duplicate detectors. That way if one goes down, I have a backup for security reasons and speed in response times.

I'm an IT guy as well so if there is any testing your want done or anything I could help with just let me know. And thanks again for a kick-ass app!

LordNex commented 2 years ago

@jakowenko I'll add this and see if it still occurs.

detectors:
  deepstack:
    timeout: 30000

In the meantime I'll look into CompreFace install on VMWare ESXi 7. But yes if we could have duplicates of the same kind of detector and maybe the option of either to run in tandem for speed or fail over for fault tolarance would be a nice feature.

cry4r commented 2 years ago

Hi, i have the same problem and on the Double take log i read this: "22-01-03 23:33:40 error: Error: connect ECONNREFUSED 192.168.0.40:80 at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)" i have a Home assistant vm on Proxmox with CPU type setted on Host.

cry4r commented 2 years ago

i tried, but without succes. I think the issue is DeepStack or DoubleTake on HAos, because i installed Deepstack on Windows10 and DoubleTake connect on this. In all case i haven't matches face on DoubleTake interface, but i have person detected event on Frigate.

Calimerorulez commented 2 years ago

Same issue here, running a LXC Proxmox container, Deepstack won't connect, but using a VM, with the 'host' type CPU configured works fine.

I would like to use an LXC, because of lower resources...

terryhonn commented 2 years ago

Same issue here, running a LXC Proxmox container, Deepstack won't connect, but using a VM, with the 'host' type CPU configured works fine.

I would like to use an LXC, because of lower resources...

Reporting same situation here, in case this ever gets any traction

Ulysses-101 commented 2 years ago

I have the same issue, both Double-Take and Deepstack are running in docker containers on my Nano Jetson. I can see DS gets requests from DT but won't connect.

[GIN] 2022/06/29 - 11:47:56 | 500 | 1m0s | 172.17.0.1 | POST "/v1/vision/face/recognize"

Any help would be greatly appricated.

lunarok commented 2 years ago

I have the same issue as @Ulysses-101, DeepStack, Frigate and Double-Take are on Jetson Nano. So no Intel KVM and it's not a "virtualisation on virtualisation" as it's pure docker directly on the Nano. But I see that our logs for Deepstack are showing 500 code and not 200. Meaning something wrong happens on Deepstack.

Looking briefly on google, I can see some errors on Deepstack running on Jetson when calling models with issue on arm64.

Don't know if it helps, at least it can be a direction.

Calimerorulez commented 2 years ago

@Ulysses-101 @lunarok Does the Nano Jetson CPU have AVX capabilities?

lunarok commented 2 years ago

@Ulysses-101 @lunarok Does the Nano Jetson CPU have AVX capabilities?

From what I found, AVX is only for INtel/AMD Jetson is ARM based, so no AVX

https://forums.developer.nvidia.com/t/sse-and-avx2-on-xavier/129546

LordNex commented 2 years ago

Are you guys using An MQTT Broker? I haven't plugged all my stuff back in yet since the upgrade but if I remember right, DT passes the image over to DS using MQTT.

The way I had mine setup was Double Take setup on an install of Home Assistant with its MQTT broker running, I then had a RPi4 running frigate with a coral TPU attached for base object detection and NVR abilities, and then DeepStack running in Docker Compose on the Jetson Nano. Double Take pulls all that together through MQTT so you might try taking a look there.

Do a search for an MQTT explorer for your computer and that should give you a bit more insight on how your devices are talking to each other in a pub/sub fashion.

lunarok commented 2 years ago

DS is not speaking MQTT, only HTTP API. MQTT part between DT and Frigate looks fine, Frigate is show in green and can see topic populated for camera found Btw, DT is using the topic subject provided in conf (mine : doubletake) for camera, but for selfstate topics is still using default topic (double-take, with -)

Error 500 show that there is an issue between DT and DS

lunarok commented 2 years ago

I just install DS on my NUC (same in docker compose) and this one is ok, DT connects to it and show green

Calimerorulez commented 2 years ago

That was my problem too, I am using Proxmox, and I had to expose the host CPU and not a virtual CPU to the container in order to get it to work, because of the AVX flag of the CPU I was using. If Deepstack does not 'see' the AVX flag, DoubleTake won't show 'green'.

LordNex commented 2 years ago

Anyone else having any luck with this. I'm just setting my stuff back up and got my Jetson Nano back running but again can't get DT to connect to it. Although it connects fine to my COU version of DeepStack running inside Home Assistant

heisenberg2980 commented 2 years ago

I had to expose the host CPU and not a virtual CPU to the container

@Calimerorulez how did you expose the host CPU to the container?

Calimerorulez commented 2 years ago

@Heisenberg2980 In Proxmox select your VM, then Hardware -> Processors -> Edit -> Type: host

Calimerorulez commented 2 years ago

@Heisenberg2980 But that is for a VM specifically, I can't recall yet how to do it for a container. I don't think that's possible, because containers always see the host CPU as it is.

heisenberg2980 commented 2 years ago

Same issue here, running a LXC Proxmox container, Deepstack won't connect, but using a VM, with the 'host' type CPU configured works fine.

I would like to use an LXC, because of lower resources...

Same issue here, Double Take cannot connect to Deepstack running on LXC Proxmox container returning the error "Face endpoints not activated".

@Calimerorulez I just ran command lscpu | grep avx in both the container and the host, and it returns blank, so I guess my server doesn´t have AVX capabilities, and if that is a requirement for Double Take to find Deepstack, that would explain why it is not connecting

marcinprzybysz86 commented 2 years ago

Same issue here, running a LXC Proxmox container, Deepstack won't connect, but using a VM, with the 'host' type CPU configured works fine. I would like to use an LXC, because of lower resources...

Same issue here, Double Take cannot connect to Deepstack running on LXC Proxmox container returning the error "Face endpoints not activated".

@Calimerorulez I just ran command lscpu | grep avx in both the container and the host, and it returns blank, so I guess my server doesn´t have AVX capabilities, and if that is a requirement for Double Take to find Deepstack, that would explain why it is not connecting

I have avx but still no luck connecting when everything is on docker @ lxc.