search

PhotonVision / photonvision

PhotonVision is the free, fast, and easy-to-use computer vision solution for the FIRST Robotics Competition.
https://photonvision.org
GNU General Public License v3.0
281 stars 203 forks source link

Calibration causes camera to hang #1032

Open agasser opened 1 year ago

agasser commented 1 year ago

While attempting to perform camera calibration, the camera stops responding in v2024.1.1-beta-3.1.

To Reproduce Steps to reproduce the behavior:

  1. Flash a fresh image on an SD card, enable the OV9281 overlay, configure team number and static IP
  2. Go to the Camera page
  3. Start calibration
  4. Move the checkerboard around in front of the camera in an attempt to detect the checkerboard

Log photonvision-journalctl.txt

Platform:

Additional context Adjusting the exposure and brightness during calibration don't see to take effect. I was not able to get the checkerboard to be detected before the camera hung. I attempted calibration multiple times, reflashed the SD card and started over, tried different cables, and even used two different cameras.

srimanachanta commented 1 year ago

I don't see anything in logs regarding calibration starting, stoping, or hanging. I do see this but this may or may not be related to your issue.

Jan 31 20:57:58 photonvision java[361]: [0:20:31.713188468] [636] ERROR RPI raspberrypi.cpp:1875 Unicam has timed out!
Jan 31 20:57:58 photonvision java[361]: [0:20:31.713204487] [636] ERROR RPI raspberrypi.cpp:1876 Please check that your camera sensor connector is attached securely.
Jan 31 20:57:58 photonvision java[361]: [0:20:31.713219135] [636] ERROR RPI raspberrypi.cpp:1877 Alternatively, try another cable and/or sensor.

Its possible that the logs weren't correctly captured, try to check if other logs were exported or try to export them again.

agasser commented 1 year ago

Do you have more troubleshooting ideas? Like I said, I tried with multiple fresh installs, multiple cables, and even two cameras. The camera works to detect 2d AprilTags, but when I try to calibrate it for 3d, the camera hangs.

srimanachanta commented 1 year ago

Do you have more troubleshooting ideas? Like I said, I tried with multiple fresh installs, multiple cables, and even two cameras. The camera works to detect 2d AprilTags, but when I try to calibrate it for 3d, the camera hangs.

Yeah, I wanna trace the cause of the hanging. I need some sort of logs to see why the program is hanging.

mcm001 commented 1 year ago

Yeah, I wanna trace the cause of the hanging. I need some sort of logs to see why the program is hanging.

That's what Journalctl logs should show

agasser commented 1 year ago

I was able to get all of the resolutions calibrated, but it took a couple tries. I don't know if it's related, but I decided I didn't know the state of the RPi firmware, so I installed the latest Rasberry Pi OS and ran sudo apt full-upgrade just to be sure everything was up to date, and then flashed a fresh PV image. After doing that, it's more stable, but the camera is still hanging.

Sorry, there isn't any more interesting information in the log. This is the latest log tail extract when the camera hung. This time it wasn't during calibration, it was running an AprilTag pipeline:

Jan 31 21:18:21 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:21 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:21 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:24 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:24 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:24 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:25 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:25 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:29 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:29 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:29 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:32 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:32 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:34 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:34 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:35 photonvision java[943]: [2023-01-31 21:18:35] [General - ConfigManager] [DEBUG] Saving to disk...
Jan 31 21:18:35 photonvision java[943]: [2023-01-31 21:18:35] [Config - SqlConfigProvider] [DEBUG] Saving to disk
Jan 31 21:18:35 photonvision java[943]: [2023-01-31 21:18:35] [Config - SqlConfigProvider] [INFO] Settings saved!
Jan 31 21:18:37 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:37 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:39 photonvision java[943]: [0:37:20.474074663] [1052]  WARN V4L2 v4l2_videodevice.cpp:2007 /dev/video0[38:cap]: Dequeue timer of 1000000.00us has expired!
Jan 31 21:18:39 photonvision java[943]: [0:37:20.474150793] [1052] ERROR RPI raspberrypi.cpp:1875 Unicam has timed out!
Jan 31 21:18:39 photonvision java[943]: [0:37:20.474167515] [1052] ERROR RPI raspberrypi.cpp:1876 Please check that your camera sensor connector is attached securely.
Jan 31 21:18:39 photonvision java[943]: [0:37:20.474183052] [1052] ERROR RPI raspberrypi.cpp:1877 Alternatively, try another cable and/or sensor.
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - MetricsManager] [DEBUG] Publishing Metrics...
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Executing "sed 's/.\{3\}$/.&/' /sys/class/thermal/thermal_zone0/temp"
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Executing "top -bn1 | grep "Cpu(s)" | sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | awk '{print 100 - $1}'"
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Executing "if   ((  $(( $(vcgencmd get_throttled | grep -Eo 0x[0-9a-fA-F]*) & 0x01 )) != 0x00 )); then echo "LOW VOLTAGE";  elif ((  $(( $(vcgencmd get_throttled | grep -Eo 0x[0-9a-fA-F]*) & 0x08 )) != 0x00 )); then echo "HIGH TEMP";  elif ((  $(( $(vcgencmd get_throttled | grep -Eo 0x[0-9a-fA-F]*) & 0x10000 )) != 0x00 )); then echo "Prev. Low Voltage";  elif ((  $(( $(vcgencmd get_throttled | grep -Eo 0x[0-9a-fA-F]*) & 0x80000 )) != 0x00 )); then echo "Prev. High Temp";  else echo "None"; fi"
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Executing "uptime -p | cut -c 4-"
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Executing "awk '/MemAvailable:/ {print int($2 / 1000);}' /proc/meminfo"
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Executing "vcgencmd get_mem malloc | grep -Eo '[0-9]+'"
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Executing "df ./ --output=pcent | tail -n +2"
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [WebServer - Server] [DEBUG] Handled HTTP request of type POST from endpoint /api/utils/publishMetrics for host 10.70.28.2 in 355.27408 ms
Jan 31 21:18:53 photonvision java[943]: [2023-01-31 21:18:53] [General - ShellExec] [DEBUG] Executing "journalctl -u photonvision.service > /tmp/photonvision-journalctl15267957074382749562.txt"

I was having this problem last week, so I ordered a new camera. This is a new camera and cable, fresh out of the box and it's still occurring.

srimanachanta commented 1 year ago

I was able to get all of the resolutions calibrated, but it took a couple tries. I don't know if it's related, but I decided I didn't know the state of the RPi firmware, so I installed the latest Rasberry Pi OS and ran sudo apt full-upgrade just to be sure everything was up to date, and then flashed a fresh PV image. After doing that, it's more stable, but the camera is still hanging.

Sorry, there isn't any more interesting information in the log. This is the latest log tail extract when the camera hung. This time it wasn't during calibration, it was running an AprilTag pipeline:

Jan 31 21:18:21 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:21 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:21 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:24 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:24 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:24 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:25 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:25 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:29 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:29 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:29 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:32 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:32 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:34 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:34 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:35 photonvision java[943]: [2023-01-31 21:18:35] [General - ConfigManager] [DEBUG] Saving to disk...
Jan 31 21:18:35 photonvision java[943]: [2023-01-31 21:18:35] [Config - SqlConfigProvider] [DEBUG] Saving to disk
Jan 31 21:18:35 photonvision java[943]: [2023-01-31 21:18:35] [Config - SqlConfigProvider] [INFO] Settings saved!
Jan 31 21:18:37 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:37 photonvision java[943]: no framebuffer, skipping
Jan 31 21:18:39 photonvision java[943]: [0:37:20.474074663] [1052]  WARN V4L2 v4l2_videodevice.cpp:2007 /dev/video0[38:cap]: Dequeue timer of 1000000.00us has expired!
Jan 31 21:18:39 photonvision java[943]: [0:37:20.474150793] [1052] ERROR RPI raspberrypi.cpp:1875 Unicam has timed out!
Jan 31 21:18:39 photonvision java[943]: [0:37:20.474167515] [1052] ERROR RPI raspberrypi.cpp:1876 Please check that your camera sensor connector is attached securely.
Jan 31 21:18:39 photonvision java[943]: [0:37:20.474183052] [1052] ERROR RPI raspberrypi.cpp:1877 Alternatively, try another cable and/or sensor.
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - MetricsManager] [DEBUG] Publishing Metrics...
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Executing "sed 's/.\{3\}$/.&/' /sys/class/thermal/thermal_zone0/temp"
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Executing "top -bn1 | grep "Cpu(s)" | sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | awk '{print 100 - $1}'"
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Executing "if   ((  $(( $(vcgencmd get_throttled | grep -Eo 0x[0-9a-fA-F]*) & 0x01 )) != 0x00 )); then echo "LOW VOLTAGE";  elif ((  $(( $(vcgencmd get_throttled | grep -Eo 0x[0-9a-fA-F]*) & 0x08 )) != 0x00 )); then echo "HIGH TEMP";  elif ((  $(( $(vcgencmd get_throttled | grep -Eo 0x[0-9a-fA-F]*) & 0x10000 )) != 0x00 )); then echo "Prev. Low Voltage";  elif ((  $(( $(vcgencmd get_throttled | grep -Eo 0x[0-9a-fA-F]*) & 0x80000 )) != 0x00 )); then echo "Prev. High Temp";  else echo "None"; fi"
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:51 photonvision java[943]: [2023-01-31 21:18:51] [General - ShellExec] [DEBUG] Executing "uptime -p | cut -c 4-"
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Executing "awk '/MemAvailable:/ {print int($2 / 1000);}' /proc/meminfo"
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Executing "vcgencmd get_mem malloc | grep -Eo '[0-9]+'"
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Executing "df ./ --output=pcent | tail -n +2"
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [General - ShellExec] [DEBUG] Got exit code 0
Jan 31 21:18:52 photonvision java[943]: [2023-01-31 21:18:52] [WebServer - Server] [DEBUG] Handled HTTP request of type POST from endpoint /api/utils/publishMetrics for host 10.70.28.2 in 355.27408 ms
Jan 31 21:18:53 photonvision java[943]: [2023-01-31 21:18:53] [General - ShellExec] [DEBUG] Executing "journalctl -u photonvision.service > /tmp/photonvision-journalctl15267957074382749562.txt"

I was having this problem last week, so I ordered a new camera. This is a new camera and cable, fresh out of the box and it's still occurring.

I have this exact camera and platform so I can try and see if I can replicate later.

amquake commented 1 year ago

I have had the exact same problem with the same hardware setup, not specifically calibrating but in general having it randomly Unicam has timed out! after some time.

mcm001 commented 1 year ago

I have had the exact same problem with the same hardware setup, not specifically calibrating but in general having it randomly Unicam has timed out! after some time.

Is this a thing that started happening this year? I haven't upgraded the base raspberry pi image yet, so I would be surprised if thats to blame

srimanachanta commented 1 year ago

No, I had the same issue but it was resolved after I reseated the cable a couple of times