search

johnolafenwa / DeepStack

The World's Leading Cross Platform AI Engine for Edge Devices
Apache License 2.0
675 stars 105 forks source link

Jetson Nano 2GB - Deepstack Face not responding after few API request #70

Open dimitribellini opened 3 years ago

dimitribellini commented 3 years ago

Dear DeepStack Team, Thanks for great software, i have bought a Nvidia Jetson Nano 2GB to test it but i have some problem with Face Detection. At the moment i have used the following docker image and enviroment:

docker run -d --runtime nvidia --gpus all -e VISION-FACE=True -e MODE=High -p 80:5000 deepquestai/deepstack:jetpack-2020.12

The problem comes after two face detection runs, the Jetson seems going on SWAP mode and the system is not responsible at all, i have wait 30 minutes to see if something was elaborated but nothing. As you suggest in most of the thread i have look on the log but is blank, no errors. I have tested the latest docker images (jetpack-2020.12,jetpack,jetpack-x3-beta) but without so much success. Have you some idea? 2GB of RAM is not enough? What does it mean “MODE=Low/Medium/High”? From the doc is not so clear :slight_smile: “High” means High CPU/RAM power usage (Better inference)?

I have made some more test using an USB3 SSD drive and remove the ZSWAP, the system is going better and the docker deepstack is working much better but after some more API request the system is going in stale with KSWAPD near 100% of CPU. The problem i think is related to the 2GB of Ram.....

Thanks so much for the answers.

dimitribellini commented 3 years ago

I have catch the error log before the crash of my Jetson Nano: kernel_panic_jetston_nano_2gb.log

dimitribellini commented 3 years ago

I have also found that on nvidia forum, maybe the problem is related to the RAM used for GPU... https://forums.developer.nvidia.com/t/general-question-about-jetsons-gpu-cpu-shared-memory-usage/61091/6

dimitribellini commented 3 years ago

Other good link https://forums.developer.nvidia.com/t/script-killed/76716/6

robmarkcole commented 3 years ago

I believe MODE=High is not supported on Jetson, remove this

dimitribellini commented 3 years ago

@robmarkcole the option is working because with Low i have worst confidence detection compared to the Mode=High

jodur commented 3 years ago

I also have problems with the Jetson 2GB for face detection. When running with the docker container with environment variable VISION-FACE=True, the service is started, but the device becomes inresponsive. The object detection doesn’t have any problems

johnolafenwa commented 3 years ago

Hello @dimitribellini @jodur , as you pointed out, this is a limitation on the memory. Generally, 4GB RAM is needed at minimum to run DeepStack. For the Jetson nano, Object Detection and scene recognition should work fine at low mode. The FACE endpoint uses more memory.

dimitribellini commented 3 years ago

Hi @johnolafenwa, thanks so much for your reply. So the Nvidia Jetson Nano 2GB is not support for Face recognition? There is no way to fix it (lower accuracy)? Thanks

CRCinAU commented 2 years ago

I just stumbled across this - perhaps it should be mentioned in the Jetson Nano documentation....

Object recognition works fine, but enabling face as well runs out of memory... This is even on the 4Gb nano...

LordNex commented 2 years ago

I have a 4 gig nano and it had been working great with Double Take, Frigate, and Home Assistant. All on separate boxes. But lately, i cannot get to the jetson by ip address at all. If I try and connect by its netbios name, it takes a min or two but finally connects. It looks like the container is running and everything looks fine. But I cannot send any data to the API end point or do anything with it. Thaats even if I create a brand new container and run it with default settings

LordNex commented 2 years ago

Any update on this? My Jetson Nano is just sitting there for now. I went ahead and just installed the Home Assistant DeepStack Addon. but it cant make use of any of the decoding abilities. It seems like it still processed requests but comes up with a Connection Error in Double Take now when the day before it didn't and nothing that I know of had changed. This is more than likely an issue with Double Take, but the combo of DeepStack, Frigate, and Double Take all integrated into Home Assistant is so very badass!

hb020 commented 2 years ago

Might I mention that Jetpack on Jetson Nano is a dead end: https://developer.nvidia.com/embedded/develop/roadmap and https://forums.developer.nvidia.com/t/jetson-software-roadmap-for-2h-2021-and-2022/177724

And be it deepstack face detection or deepstack with custom models like the combined model from MikeLud/DeepStack-Security-Camera-Models, even the 4GB version quickly runs out of memory and becomes unresponsive.

Time for another HW accelerator. Coral would be ideal (just hoping...)

LordNex commented 2 years ago

I’m running the 4 gig Nano and it does an ok job. But even with no load it has connection issues. Of course my use case might be different. I have the Nano running just DeepStack in docker, frigate running on a rpi4 8gig, and Home Assistant with the Double Take addin. I have a coral TPU coming but I was planning on just putting that on the RPi for frigate to use for fast object detection. Then if it sees a face it sends it through mqtt to DeepStack to be processed. This way DeepStack isn’t having to monitor feeds or anything. Just listen to the mqtt topic.

Do you think it would be better to run both frigate and DeepStack on the Nano once I get the coral? I could add an M2 card or SSD drive that it can use as a RAM disk.

Benjamin T Powers

On Dec 30, 2021, at 11:50 AM, hb020 @.***> wrote:



Might I mention that Jetpack on Jetson Nano is a dead end: https://developer.nvidia.com/embedded/develop/roadmap and https://forums.developer.nvidia.com/t/jetson-software-roadmap-for-2h-2021-and-2022/177724

And be it deepstack face detection or deepstack with custom models like the combined model from MikeLud/DeepStack-Security-Camera-Models, even the 4GB version quickly runs out of memory and becomes unresponsive.

Time for another HW accelerator.

— Reply to this email directly, view it on GitHubhttps://github.com/johnolafenwa/DeepStack/issues/70#issuecomment-1003125800, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACGXNJFFXC3UOZWIFDITTDTUTSLVLANCNFSM4WAWI5YA. You are receiving this because you commented.Message ID: @.***>

hb020 commented 2 years ago

Sorry, I don't run Frigate: am using BlueIris for now, so cannot comment on that.

About the RAM disk: a RAM disk is only possible if you have extra RAM, which you don't have on the Nano. If you want to make more memory available to your system, you will need a fast swap partition or swap file on a flash drive. Adding RAM is not possible. For swap, you have 3 options:

In essence: probably a waste of time.

LordNex commented 2 years ago

What do you suggest? And yea I meant swap file on a fast SSD. Don’t know why I said RAM disk. I have a pretty beefy server with 12 core i7 and 32 gigs of ram. It’s running ESXi 7 and basically is just for OpenMediaVault (I have a lot of drives in the full size chassis) and a Gaming server setup through Amp on Ubuntu Server. Currently it’s just a Minecraft server for my son and me and a few friends. But as soon as my Starlink comes I’m all set to start some serious game hosting. Have a huge home network with a Fortigate Firewall. So it’ll be done right.

My use case it for facial recognition on cameras we have on our property. It basically says that if it sees one of the family at the door and some other checks and balances thrown in, to unlock the door, turn on lights if it’s dark, and greet the person with a custom greeting tailored to that person. Basically I have all that working but as you probably well know that as you start adding more cameras, you need more power. I thought the Jetson would be enough but I bought it over a year ago before a family member passed so I just got back to it a few weeks ago. I got tired of having issues and looked and though about a coral dev board but figured I’d run into the same issue I did with the Nano. So I opted for the usb coral that way I can just plug it into whatever I need.

So like I said any suggestions you have would be welcome. The one reason I didn’t want to go blue iris is I didn’t want to eat up another windows license if I didn’t need too. Which is why I usually opt for Linux distros if available and secure. Thanks for all the help and for writing all this for free!

Benjamin T Powers

On Dec 31, 2021, at 12:28 AM, hb020 @.***> wrote:



Sorry, I don't run Frigate: am using BlueIris for now, so cannot comment on that.

About the RAM disk: a RAM disk is only possible if you have extra RAM, which you don't have on the Nano. If you want to make more memory available to your system, you will need a fast swap partition or swap file on a flash drive. Adding RAM is not possible. For swap, you have 3 options:

In essence: probably a waste of time.

— Reply to this email directly, view it on GitHubhttps://github.com/johnolafenwa/DeepStack/issues/70#issuecomment-1003284457, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACGXNJEZ6ACPLLWHTNQJE7DUTVERRANCNFSM4WAWI5YA. You are receiving this because you commented.Message ID: @.***>

hb020 commented 2 years ago

Suggestions... Well, depends.

Can you add a GPU to your ESXi? With PCI Passthrough, you could make deepstack use it. Uses a lot more power though.

Is your internet link reliable enough to make your face detection depend on its availability? Then run your models in the cloud. Many options there, some for a low fee. Don't know about AWS, as that is strictly out of bounds in the field I work in, but GCP has some nice options.

About WIndows: same considerations as you: I prefer Linux. The only Win machine at home is for BI, but I managed to find an old Win7 license that I upgraded to Win10, so I'm fine. You might also be able to find one.

Although adding RAM to the Jetson Nano module is not possible (it is on the module, not on the carrier board), you do have the option of choosing a better carrier board, like the Auvidia JN-30PD or JN30-PSE, that has full nvme possibilities. Hence allowing faster flash storage. But I cannot comment more on its capabilities, as with the Jetson Nano software support nearing EOL, I have abandoned all research related to that device.

A warning about Coral: the most flexible is indeed still the USB version, that unfortunately is hard to get working under VMWare (be it ESXi or Fusion), so needs either proxmox, a dedicated host (ex.: raspi with > 3A PSU), or a dedicated PCI passed through USB controller on ESXi.

Deepstack + Coral is by far my preferred solution, but not possible (yet? ever?).