Closed kevinmilner closed 1 year ago
If it happens on the host OS as well as in a VM then I'd say it is most likely a hardware issue, not sure if it is the adapter but I've not seen this behavior much at all with PCIe based corals.
If it happens on the host OS as well as in a VM then I'd say it is most likely a hardware issue, not sure if it is the adapter but I've not seen this behavior much at all with PCIe based corals.
Thanks for the reply. I tried a different PCIe slot but same result. One more data point, I tailed the logs and dmesg at the same time and discovered some things about the timing of the various log messages. At the point of this log:
frigate.watchdog INFO : Detection appears to be stuck. Restarting detection process...
...the only relevant dmesg messages were:
[ 284.222972] x86/PAT: frigate.detecto:59726 map pfn RAM range req uncached-minus for [mem 0x10041c000-0x10041ffff], got write-back
[ 292.125253] apex 0000:02:00.0: Apex performance not throttled due to temperature
[ 297.245143] apex 0000:02:00.0: Apex performance not throttled due to temperature
That "not throttled" message keeps repeating for a while, then the RAM errors don't hit until after this log line:
root INFO : Detection process didnt exit. Force killing...
That line is followed a few seconds later in dmesg
by:
[ 347.469109] apex 0000:02:00.0: RAM did not enable within timeout (12000 ms)
[ 347.476731] apex 0000:02:00.0: Apex performance not throttled due to temperature
[ 360.361156] apex 0000:02:00.0: RAM did not enable within timeout (12000 ms)
[ 360.368172] apex 0000:02:00.0: Error in device open cb: -110
So maybe the RAM error is a symptom of the detection processing being force killed and might not itself be a hardware issue? Are there any configuration issues that can cause the detection process to stall like that? Things like improper resolution, ffmpeg flags, etc? This is my first go at things, so it could definitely be wrong. I'll try to get a simple test case working with a CPU detector first to verify that's not the issue.
Like I said I've not seen this with PCIe based coral. There are a handful of cases of this error happening with the USB coral and in those cases it was:
So, for a PCIe coral it's not clear but I really doubt it has anything to do with ffmpeg or frigate itself
OK, thanks for the info. When you say "this error happening" are you referring to the log message about how Detection appears to be stuck.
, the dmesg about Apex performance not throttled due to temperature
, and/or the dmesg about RAM did not enable within timeout (12000 ms)
? Is that not throttled due to temperature
a common message or an indication of a problem (I know the other 2 are, but not sure about that one)?
My CPU-only test did fire up just fine, so I think you're right about it being hardware. I'm just trying to understand what's normal and what's not in the logs while I go down the hardware debugging rabbit hole. Step 1 will be a new adapter since that's a lot easier to get than a new TPU.
Really appreciate the time and the quick responses!
OK, thanks for the info. When you say "this error happening" are you referring to the log message about how
Detection appears to be stuck.
, the dmesg aboutApex performance not throttled due to temperature
, and/or the dmesg aboutRAM did not enable within timeout (12000 ms)
? Is thatnot throttled due to temperature
a common message or an indication of a problem (I know the other 2 are, but not sure about that one)?
I am referring to detection being stuck.
Haven't figure this out yet, but I can confirm that it's a coral issue and not related to frigate, closing issue.
For anyone who finds this, I'll be trying to figure this out over on the coral issue tracker: https://github.com/google-coral/edgetpu/issues/741#event-8939904810
FWIW, I ran into the same issue on my machine running the A+E key Coral TPU using a PCIe card "converter". I am running frigate in AlmaLinux 9.2 via rootless Podman 4.4.1. On the host I am running the gasket-dkms
module.
Digging around I found this which did solve the issue for me https://github.com/google-coral/edgetpu/issues/345#issuecomment-1656841523
I already had the pcie_aspm=off
kernel option configured, but it was removing and rescanning that seemed to "reset" the Coral and allowed the container detect the TPU once again.
I had the same error message and it was a reproducible issue for me related to temperature. I had been using the PCIe TPU reliably with a single core. Once I added the Dual TPU B+M adapter key and started leveraging both cores, temperatures would climb to 100 and then the errors would start.
The driver has an automatic full shutdown by default at 90 I believe so that would make sense. I have my dual corals cooled via 40mm noctua fan.
Describe the problem you are having
Upon startup, the PCIe TPU is found, but then the detector is reported as stuck, is killed, and never comes back to life. I am running this as an app on a Truenas Scale (bluefin) using Truecharts (appologies for the verbose auto-generated config file below). I have the drivers successfully installed on the host system.
My computer (an HP Z230 workstation) doesn't have any mPCIe slots, so I am using the Ableconn PEX-MP117 adapter; many people on Amazon have reported success using with this adapter with the Coral.
I have also tried passing the PCIe card through to a VM and using a simpler config with docker-compose, but got the same errors (I originally thought they were due to PCIe passthrough issues, but them occurring on the bare metal install makes me think otherwise). When I was doing tests with the VM, I did not have the drivers installed on the host (only the VM). When switching to bare metal, I disabled VM PCIE passthrough so the VM shouldn't be mucking with things.
Version
0.11.1-2eada21
Frigate config file
docker-compose file or Docker CLI command
Relevant log output
Operating system
Debian
Install method
Docker Compose
Coral version
PCIe
Any other information that may be helpful
I see the following in
dmesg
. Same exact error messages when run on bare metal as when I tried it inside of a VM: