gopro / labs

GoPro Labs
Apache License 2.0
495 stars 40 forks source link

Random restarts #915

Closed tony-gutierrez closed 1 month ago

tony-gutierrez commented 6 months ago

I am using opengopro commands to configure 4 cameras, record a timewarp of about 90 seconds, DL it to a local machine, delete it , and repeat. Cameras are powered via a usb hub, no batteries. Using gplabs trust usb and wake on power.

This can run fine for 40 or so iterations, but at some point the cameras reboot in some manner. They revert to default recording settings. The reboot always happens during the video file download, which will then time out. I have retry logic, and often the file will DL fine on the second attempt after the reboot. This happens on all cams (~20 in service) occasionally.

I tried the undocumented !MDBGL=1 and this is what is in the limited logs. The recording video is timestamped 15:25.


[2024-05-16 15:22:21 00%   ] GP2 Temp.:61 
[2024-05-16 15:22:32 00%   ] GP2 Temp.:62 
[2024-05-16 15:22:41 00%   ] GP2 Temp.:64 
[2024-05-16 15:23:31 00%   ] GP2 Temp.:65 
[2024-05-16 15:24:01 00%   ] GP2 Temp.:61 
[2024-05-16 15:24:12 00%   ] GP2 Temp.:62 
[2024-05-16 15:24:21 00%   ] GP2 Temp.:64 
[2024-05-16 15:24:51 00%   ] GP2 Temp.:65 
[2024-05-16 15:25:41 00%   ] GP2 Temp.:62 
[2024-05-16 15:26:01 00%   ] GP2 Temp.:64 
[2024-05-16 15:26:41 00%   ] GP2 Temp.:65 
[2024-05-16 15:27:26-delayed ] wake reason:HEARTBEAT_TIMEOUT 
[2024-05-16 15:27:27 --%   ] Labs:Display Messages 
[2024-05-16 15:27:29 00%   ] display 2000ms:Features activated: 
TUSB,WAKE=2
[2024-05-16 15:27:31 00%   ] Features activated: 
TUSB,WAKE=2
[2024-05-16 15:27:31 00%   ] display 1000ms:GoPro Labs
Beta Firmware, Participation Terms Apply 
[2024-05-16 15:27:31 00%   ] Labs:Main Start 
[2024-05-16 15:27:31 00%   ] Labs:H23.01.02.20.70 
[2024-05-16 15:27:31 00%   ] Labs LCD ID:195 
[2024-05-16 15:27:31 00%   ] Naked when 0:1 
[2024-05-16 15:27:31 00%   ] Labs:Initialize 
[2024-05-16 15:27:31 00%   ] Bat Temp.:0 
[2024-05-16 15:27:31 00%   ] delay.txt:Not found 
[2024-05-16 15:27:42 00%   ] GP2 Temp.:58 
[2024-05-16 15:28:01 00%   ] GP2 Temp.:57 
[2024-05-16 15:28:21 00%   ] GP2 Temp.:56 
[2024-05-16 15:28:41 00%   ] GP2 Temp.:55 
[2024-05-16 15:29:11 00%   ] GP2 Temp.:54 
[2024-05-16 15:29:41 00%   ] GP2 Temp.:53 
[2024-05-16 15:30:31 00%   ] GP2 Temp.:52 
[2024-05-16 15:31:11 00%   ] GP2 Temp.:51 
[2024-05-16 15:32:21 00%   ] GP2 Temp.:50 
[2024-05-16 15:33:31 00%   ] GP2 Temp.:49 
[2024-05-16 15:34:51 00%   ] GP2 Temp.:48 
[2024-05-16 15:37:11 00%   ] GP2 Temp.:47 
[2024-05-16 15:41:01 00%   ] GP2 Temp.:46 
[2024-05-16 15:41:12 00%   ] GP2 Temp.:47 
[2024-05-16 15:41:23 00%   ] GP2 Temp.:46 
[2024-05-16 15:46:51 00%   ] GP2 Temp.:45 
[2024-05-16 15:47:02 00%   ] GP2 Temp.:46 
[2024-05-16 15:47:21 00%   ] GP2 Temp.:45 
[2024-05-16 15:47:32 00%   ] GP2 Temp.:46 
[2024-05-16 15:47:51 00%   ] GP2 Temp.:45 
[2024-05-16 15:48:12 00%   ] GP2 Temp.:46 
[2024-05-16 15:48:41 00%   ] GP2 Temp.:45 
[2024-05-16 17:33:21 00%   ] GP2 Temp.:44 
[2024-05-16 17:33:32 00%   ] GP2 Temp.:45 
[2024-05-16 17:33:43 00%   ] GP2 Temp.:44 
[2024-05-16 17:52:21 00%   ] GP2 Temp.:43 
[2024-05-16 17:52:32 00%   ] GP2 Temp.:44 
[2024-05-16 17:53:01 00%   ] GP2 Temp.:43 
[2024-05-16 17:53:21 00%   ] GP2 Temp.:44 
[2024-05-16 17:53:51 00%   ] GP2 Temp.:43 
[2024-05-16 17:54:11 00%   ] GP2 Temp.:44 
[2024-05-16 17:54:22 00%   ] GP2 Temp.:43 
[2024-05-16 17:55:02 00%   ] GP2 Temp.:44 
[2024-05-16 17:55:13 00%   ] GP2 Temp.:43 
dnewman-gpsw commented 6 months ago

The heart beat timeout is a self reboot as the camera was stuck waiting too long for something. This is not likely a Labs issue, as you aren't running any Labs commands (in a loop), so I'm guessing this would also happen on stock firmware. Have you ask the question to the open gopro team?

pursual commented 6 months ago

The open GoPro team does not seem to put much effort into issues they consider firmware related, and I'm not using their SDK, as it was buggy and limited and moving too slow.

Is there any way to get more detail out of the camera about what exactly is happening? I see an mdb12.db file, but I'm not sure what format it is in.

I'll try testing on stock firmware.

pursual commented 6 months ago

I have noticed that key 111 in the state status is reporting that my SD card is too slow. I am using fast SD cards. Can this value be trusted? This is the card that I am using: PXL_20240517_125525547 MP

dnewman-gpsw commented 6 months ago

I don't know what key 111 is? I'm sure that card is plenty fast, although I haven't used a 1TB card. From Labs there isn't anymore information to extract, as the crash is likely related to a network error, which is happening in another operating system. Labs run on the RTOS, networking is in Linux. This is also why Labs and Open GoPro are separate. Mdb12 is propriety file used by Quik, I don't know it's format.

Odd that Open GoPro considers this is firmware issue, didn't you say it only happens during transfers? True the under lying file I/o is RTOS. Is there any Labs only that causes this? Looking for a lead to investigate.