Closed reidprichard closed 1 year ago
NickM-27
commented
1 year ago Seeing a lot of errors on the cameras which could be contributing. I don't see any reason why that would randomly happen unless the scores are just different for this model and aren't working with the default threshold set.
reidprichard
commented
1 year ago Yeah, I've noticed those errors. What's going on there? If I tune into the stream via VLC or WebRTC it is flawless.
Looking back through my event logs, it seems my recollection of it working perfectly today may have been inaccurate. While there are lots of "car" events throughout the day, there's only one "dog" event today. Perhaps yolov7-tiny is just bad at detecting dogs? I'll try again with yolov4, which I am certain worked in the past.
reidprichard
commented
1 year ago Okay, I tried yolov4-tiny-416 (which I've used on a prior TensorRT fork of Frigate with great success) and no luck. I also tried using the camera's 640x360 substream (with detect resolution adjusted to match), and that didn't work either. It will label my dog as "person" but doesn't flash "dog" on the screen for a second.
I'm not quite sure how/where the labelmap is encoded, but is it possible that got messed up somehow?
NickM-27
commented
1 year ago The labelmap is the same as the default one that is used. You may want to try lowering the threshold for dog and see if that helps.
reidprichard
commented
1 year ago Thanks. Even if it's not over threshold, shouldn't a detection box with low confidence show up in the debug view?
reidprichard
commented
1 year ago I just tried using the yolov4-tiny-416.trt that I had previous used with good success. As with yolov7, it continues to ignore my dogs or show "person" with low confidence for them.
NickM-27
commented
1 year ago You'll want to lower the min_score, that's a good point.
It's been tested and working as expected, the fact that it showed some detections for these other labels means it's working as expected (it's not like it's ignoring the object label) and most likely the model just isn't performing well in the current conditions. You could try downloading a previous event video where it worked and sending a camera with it to see how it goes
reidprichard
commented
1 year ago Okay, dropped min_area to 1000, min_score to 0.1, and threshold to 0.1. I tried with both the high-res (1920x1280) and low-res (640x360) streams - neither even briefly picked up either of my two dogs in a well-lit room. It just seems bizarre that the YOLO models would be performing so terribly in comparison to the CPU model.
I don't suppose it's possible to use different detectors for different object types? Or even both detectors in parallel indiscriminately? It'd be kind of nice to at least be able to pick up the dogs with a CPU detector on the low-res stream, then pick up finer details with the GPU on the high-res stream.
Edit: Switched back to CPU detectors with default settings, and it's picking up my dogs every time they walk by with 70-80% confidence.
NickM-27
commented
1 year ago I don't suppose it's possible to use different detectors for different object types? Or even both detectors in parallel indiscriminately? It'd be kind of nice to at least be able to pick up the dogs with a CPU detector on the low-res stream, then pick up finer details with the GPU on the high-res stream.
No that is not currently supported. It's more complicated than it may seem.
The device doing the detection has nothing to do with it, the results are due to the model that is being run. The coral and CPU run the same model with the same results. As far as TensorRT I can't say, I tested the yolov4-tiny briefly and it did see my dog but I haven't used it. May need to see if others have had similar experiences or if perhaps there's a model that works better for dogs. You may also try the non tiny model
reidprichard
commented
1 year ago I figured as much. I suppose if I really wanted to I could just have two different Frigate containers.
I didn't expect the device type was playing a role, but I assumed there were technical limitations preventing the CPU model from being run on a GPU.
I built the full list of available models last night, and today I'll see how they work. I didn't see a non-tiny v7 on the list, so I assume you're talking about one of the earlier models.
Edit: Going to make a list of my attempts and continue to update results here:
I notice "car" showing up in my event log a few times since I've been testing, so that apparently works for at least some of these models.
It seems really suspect that four different models with two different input streams haven't detected my dogs even once when the CPU detection model nails it every time. I don't want to post any images, but these are not exactly difficult scenarios for detection - we're talking my dog standing in side profile, taking up ~20% of the frame's height, with a relatively uncomplicated backdrop. Is there no chance at all Frigate isn't somehow neglecting to look for the "dog" object?
On the v7-tiny-416 and the 1080p stream, I enabled detection for "hat", "eye glasses", and "cup". I couldn't get a single detection box for a pair of glasses, baseball cap, or coffee mug (not sure if this counts as a "cup").
While I know that the model dimensions must be updated to match the TRT file, do I need to be messing with the input_tensor and input_pixel_format settings?
How does Frigate handle a model with dimensions greater than the stream? I.e., yolov4x-mish-640 on a 640x360 feed.
NickM-27
commented
1 year ago Looks like the docs need may need to be updated but this should not make a difference, you can try adding this as your labelmap to the model config: labelmap_path: /trt-models/coco_91cl.txt
reidprichard
commented
1 year ago Hmm, I don't seem to have that file in my trt-models directory. Should it have been generated by tensorrt_models.sh? Edit: Found that file on the OpenVINO Github and put it in the trt-models directory. I assume it's not an issue that OpenVINO is the Intel toolkit and I'm using TensorRT.
That didn't seem to do it. Haven't tested on the dogs yet, but I'm getting nothing for eyeglasses or cup.
If it's any help, this is the image that I previously ran that had flawless detection with GPU. Per this discussion, it was running yolov4-tiny-416. I also don't see a labelmap .txt file in that model's directory either.
NickM-27
commented
1 year ago I don't think it is necessary, I checked them both and they are the same file contents. The models are also pulled from a repo owned by the author of that discussion so I don't see why it would be any different
reidprichard
commented
1 year ago I'm doing my best to recreate the setup with yeahme49's image. Unfortunately, those YAML files have been lost to time, and most of my cameras have changed (from running Wyze's RTSP firmware to using wz_mini_hacks). I'm still not having any luck detecting anything other than person - it's hard to say whether a difference in my config or a difference in the hardware could be the issue. Although it's been long enough that now I'm not as confident in my memory that it was correctly detecting dogs on yeahme49's image.
Am I the only one having this issue? I'd think there would be a pretty good sized crowed running this on a GPU with detection other than just person.
NickM-27
commented
1 year ago I was able to test the yolo7-tiny-416 and it saw my dog as a cat. Disabling cat it did not see my dog at all.
That being said, it saw many other objects correctly (car, cat, mouse, etc)
reidprichard
commented
1 year ago Interesting - I'll give it a shot with cat enabled. Unfortunately, I don't have any mice handy :)
I'm pretty much a cretin when it comes to AI topics. Would this be the place to discuss yolo7's accuracy?
NickM-27
commented
1 year ago Interesting - I'll give it a shot with
catenabled. Unfortunately, I don't have any mice handy :)
To be clear, the mouse label in COCO dataset refers to a computer mouse, not the animal
I'm pretty much a cretin when it comes to AI topics. Would [this] be the place to discuss yolo7's accuracy?
I don't think you linked anything?
reidprichard
commented
1 year ago Ohh, that makes so much more sense. I though it funny that you had living mice handy for testing!
Oops, edited to add the link.
NickM-27
commented
1 year ago I don't think that would be the place, it would be where the models came from. But again we probably want others to confirm they see the same. I only tested on one camera
reidprichard
commented
1 year ago I enabled cat and mouse, and it is indeed occasionally picking up my dogs as cats. However, mouse is not working. I get the bounding box around it, but it thinks it is a person. Maybe that's just from my hand?
I also just updated to the latest image (9621b4b-tensorrt).
jtronicus
commented
1 year ago I am seeing the same thing. Dogs are detected as "person". I am not sure when the issue started, but it has been since at least 2/21/2023.
I know it worked a few months ago when I was using a coral device, but I dont know if it has ever worked correctly using yolov7-tiny-416.trt
reidprichard
commented
1 year ago Okay, mine finally detected one of my dogs as "dog". In the last several days, it has detected my dogs as "cat" probably 40 times, "person" maybe 5 times, and "dog" just once.
Is it possible the "cat" and "dog" labels are swapped? I would be curious for someone with a cat to report whether their cat is detected as "cat", "dog", or something else.
NickM-27
commented
1 year ago Is it possible the "cat" and "dog" labels are swapped? I would be curious for someone with a cat to report whether their cat is detected as "cat", "dog", or something else.
They have the same labelmaps which are pretty standard so that is really unlikely.
I also ran a test on a video of a cat and it only ever got detected as person or cat (pretty typical for models trained on COCO dataset). The scores are also pretty low so you may just need to adjust the min_score and threshold
jtronicus
commented
1 year ago does frigate add the labels to the image, or is it added by the coral/tensor device?
reidprichard
commented
1 year ago @NickM-27, I'll let it run with lowered thresholds for "dog" and see what happens. I suppose at this point there's not much more to discuss here if the issue lies with the models themselves.
@jtronicus, I believe the device (coral, CPU, tensor) is just executing code provided by Frigate. Frigate is responsible for supplying the model and the labelmap, which associates a model output with a word like "dog", "cat", etc.
NickM-27
commented
1 year ago @NickM-27, I'll let it run with lowered thresholds for "dog" and see what happens. I suppose at this point there's not much more to discuss here if the issue lies with the models themselves.
Yes, seems to be the case.
@jtronicus, I believe the device (coral, CPU, tensor) is just executing code provided by Frigate. Frigate is responsible for supplying the model and the labelmap, which associates a model output with a word like "dog", "cat", etc.
correct
reidprichard
commented
1 year ago I've been doing a bit more digging here. Using the YOLO7 authors' Python code from their repository, I ran YOLO7 on a capture from one of my cameras (which I manually scaled to 640p to match the model). The result was absolutely incredible - it picked out my couch, potted plants, computer monitors, books on a bookshelf, etc. Importantly, though, it caught my dog with 73% certainty.
I am an absolute noob when it comes to AI, but it seems like an important difference could be the weights used. I notice the install script uses weights (*.weights files) from the darknet repo, which I presume are different from the weights published by the YOLO7 authors (*.pt files).
I tried using the darknet weights in the YOLO7 Python script for comparison, but it seems the file types are incompatible. I suppose this makes sense, as the models in Frigate have to be generated for my specific hardware while the original YOLO7 model ran on my GPU without needing any such preparation.
I'm happy to continue investigating this but am a bit stumped (and out of time at the moment). Does any have any suggestions for next steps?
monoimoi
commented
1 year ago I am experiencing a similar issue with cuda toolkit 11.7, nvidia 515 drivers on 2 Teslas V100. Impossible to recognize anything but a person. I used both yolo 4 and 7 mini 416 with no change in the outcome. Switching to CPU detector has no such issues, it's performance is excellent.
github-actions[bot]
commented
1 year ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
cliffburdick
commented
1 year ago Please keep this one open
github-actions[bot]
commented
1 year ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Describe the problem you are having
This is the darnedest thing - I am 99.9% certain everything was working earlier today, and I don't think I've changed anything that should affect this. Using a TensorRT detector, I am unable to detect anything but "person". If I comment out the model info and replace the tensorrt detector with a pair of cpu detectors, I am able to detect other objects (namely, my dogs). Really scratching my head as to what could have changed today.
I might just roll the VM back to a snapshot earlier today when everything is working, but I thought I'd try to get to the bottom of it if possible.
Version
0.12.0-1715e2e
Frigate config file
Relevant log output
FFprobe output from your camera
Frigate stats
No response
Operating system
Other Linux
Install method
Docker Compose
Coral version
CPU (no coral)
Network connection
Wired
Camera make and model
Wyze v3
Any other information that may be helpful
Running on Ubuntu 22.04. I deleted the trt-models folder and regenerated yolov7-tiny-416.trt, but no luck.