search

raspberrypi / firmware

This repository contains pre-compiled binaries of the current Raspberry Pi kernel and modules, userspace libraries, and bootloader/GPU firmware.
5.15k stars 1.68k forks source link

Camera stopped working. assert( *p==0xa55a5aa5 ) failed #1585

Open arkanoid87 opened 3 years ago

arkanoid87 commented 3 years ago

Describe the bug camera stopped working

To reproduce can't reproduce. After reboot it seems working again, I'll let you know if this starts repeating consistently (and ends as a software bug and not hardware failure)

Expected behaviour camera keeps working

Actual behaviour camera stopped working without apparent reason. "raspinfo" and "vcgencmd get_camera" hang forever. "vcdgb log assert" show error

System

System Information
------------------

Raspberry Pi 3 Model A Plus Rev 1.0
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"

Raspberry Pi reference 2021-01-11
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 21090519d85bdaa1615d5d5057d37b09368ea5d2, stage2                                                                                                    

Linux 66cda39a 5.10.11-v7+ #1399 SMP Thu Jan 28 12:06:05 GMT 2021 armv7l GNU/Linux
Revision        : 9020e0
Serial          : 0000000066cda39a
Model           : Raspberry Pi 3 Model A Plus Rev 1.0

then hangs forever. Raspinfo doesn't quit.

vcdbg log msg

001335.782: brfs: File read: /mfs/sd/config.txt
001336.312: brfs: File read: 224 bytes
001389.474: HDMI0:EDID error reading EDID block 0 attempt 0
001390.742: HDMI0:EDID error reading EDID block 0 attempt 1
001392.006: HDMI0:EDID error reading EDID block 0 attempt 2
001393.273: HDMI0:EDID error reading EDID block 0 attempt 3
001394.539: HDMI0:EDID error reading EDID block 0 attempt 4
001395.805: HDMI0:EDID error reading EDID block 0 attempt 5
001397.069: HDMI0:EDID error reading EDID block 0 attempt 6
001398.335: HDMI0:EDID error reading EDID block 0 attempt 7
001399.600: HDMI0:EDID error reading EDID block 0 attempt 8
001400.867: HDMI0:EDID error reading EDID block 0 attempt 9
001401.889: HDMI0:EDID giving up on reading EDID block 0
001403.362: brfs: File read: /mfs/sd/config.txt
002250.947: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
002253.138: *** Restart logging
002253.165: brfs: File read: 224 bytes
002254.043: hdmi: HDMI0:EDID error reading EDID block 0 attempt 0
002255.313: hdmi: HDMI0:EDID error reading EDID block 0 attempt 1
002256.583: hdmi: HDMI0:EDID error reading EDID block 0 attempt 2
002257.853: hdmi: HDMI0:EDID error reading EDID block 0 attempt 3
002259.121: hdmi: HDMI0:EDID error reading EDID block 0 attempt 4
002260.391: hdmi: HDMI0:EDID error reading EDID block 0 attempt 5
002261.660: hdmi: HDMI0:EDID error reading EDID block 0 attempt 6
002262.930: hdmi: HDMI0:EDID error reading EDID block 0 attempt 7
002264.198: hdmi: HDMI0:EDID error reading EDID block 0 attempt 8
002265.469: hdmi: HDMI0:EDID error reading EDID block 0 attempt 9
002266.495: hdmi: HDMI0:EDID giving up on reading EDID block 0
002266.830: hdmi: HDMI0:EDID error reading EDID block 0 attempt 0
002268.100: hdmi: HDMI0:EDID error reading EDID block 0 attempt 1
002269.369: hdmi: HDMI0:EDID error reading EDID block 0 attempt 2
002270.639: hdmi: HDMI0:EDID error reading EDID block 0 attempt 3
002271.907: hdmi: HDMI0:EDID error reading EDID block 0 attempt 4
002273.178: hdmi: HDMI0:EDID error reading EDID block 0 attempt 5
002274.447: hdmi: HDMI0:EDID error reading EDID block 0 attempt 6
002275.718: hdmi: HDMI0:EDID error reading EDID block 0 attempt 7
002276.986: hdmi: HDMI0:EDID error reading EDID block 0 attempt 8
002278.256: hdmi: HDMI0:EDID error reading EDID block 0 attempt 9
002279.282: hdmi: HDMI0:EDID giving up on reading EDID block 0
002279.326: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
002279.349: HDMI0: hdmi_pixel_encoding: 162000000
002279.595: vec: vec_middleware_power_on: vec_base: 0x7e806000 rev-id 0x00002708 @ vec: 0x7e806100 @ 0x00000420 enc: 0x7e806060 @ 0x00000220 cgmsae: 0x7e80605c @ 0x00000000
002285.107: dtb_file 'bcm2710-rpi-3-a-plus.dtb'
002285.687: dtb_file 'bcm2710-rpi-3-b-plus.dtb'
002289.566: brfs: File read: /mfs/sd/bcm2710-rpi-3-b-plus.dtb
002289.591: Loading 'bcm2710-rpi-3-b-plus.dtb' to 0x100 size 0x707f
002303.048: brfs: File read: 28799 bytes
002314.665: brfs: File read: /mfs/sd/overlays/overlay_map.dtb
002384.863: brfs: File read: 1523 bytes
002386.684: brfs: File read: /mfs/sd/config.txt
002386.719: dtparam: audio=on
002397.018: dtparam: i2s=on
002404.137: dtparam: act_led_trigger=heartbeat
002415.759: dtparam: act_led_activelow=off
002427.119: dtparam: pwr_led_trigger=none
002438.793: dtparam: pwr_led_activelow=off
002456.264: brfs: File read: 224 bytes
002457.059: brfs: File read: /mfs/sd/cmdline.txt
002457.108: Read command line from file 'cmdline.txt':
002457.128: 'console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait'
004478.095: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
004533.355: brfs: File read: 115 bytes
004970.306: brfs: File read: /mfs/sd/kernel7.img
004970.330: Loading 'kernel7.img' to 0x8000 size 0x6034a8
004970.360: Device tree loaded to 0x17fe8b00 (size 0x74cf)
004973.104: gpioman: gpioman_get_pin_num: pin SDCARD_CONTROL_POWER not defined
008656.032: vchiq_core: vchiq_init_state: slot_zero = 0xd3d80000, is_master = 1
013071.742: camsubs: Looking for camera 0: i2c_port = 0, led gpio = -1, power enable gpio = 133
013372.471: camsubs: Camera not found
013372.611: camsubs: Looking for camera 0: i2c_port = 0, led gpio = -1, power enable gpio = 133
013673.330: camsubs: Camera not found
013673.479: camsubs: Looking for camera 0: i2c_port = 0, led gpio = -1, power enable gpio = 133
013675.100: camsubs: Camera found OK
023375.670: Using RPI AWB
023428.969: brfs: File read: 6304936 bytes
023429.012: mmal: mmal_vll_load: could not load VLL 'videnc.vll': 
051358.837: Using RPI AWB
051405.344: mmal: mmal_vll_load: could not load VLL 'videnc.vll': 
1899103.647: vcos_abort: Halting
1901095.809: mmal: fail_destroy: Timed out waiting to destroy ril.camera

vcdbg log assert

1899103.621: assert( *p==0xa55a5aa5 ) failed; ../../../../../vcfw/rtos/common/rtos_common_malloc.c::rtos_pool_aligned_free line 205 rev 99d9a48
vcdbg_ctx_get_dump_stack: dump_stack failed
----------------

dmesg

[443108.035046] INFO: task queue0:src:315 blocked for more than 614 seconds.
[443108.035067]       Tainted: G         C O      5.10.11-v7+ #1399
[443108.035080] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[443108.035094] task:queue0:src      state:D stack:    0 pid:  315 ppid:     1 flags:0x00000005
[443108.035123] Backtrace:
[443108.035166] [<809e5190>] (__schedule) from [<809e5b60>] (schedule+0x68/0xe4)
[443108.035189]  r10:83a6fc00 r9:00000002 r8:7fffffff r7:80e05008 r6:83a6c55c r5:8addbe00
[443108.035203]  r4:ffffe000
[443108.035226] [<809e5af8>] (schedule) from [<809e9fc4>] (schedule_timeout+0x2c8/0x36c)
[443108.035242]  r5:80e05008 r4:83a6c558
[443108.035265] [<809e9cfc>] (schedule_timeout) from [<809e7114>] (wait_for_completion+0xb8/0x138)
[443108.035287]  r9:00000002 r8:7fffffff r7:ffffe000 r6:83a6c55c r5:80e05008 r4:83a6c558
[443108.035311] [<809e705c>] (wait_for_completion) from [<808497a0>] (vchiq_release+0x120/0x2f4)
[443108.035332]  r9:80ff2b28 r8:80e05008 r7:83a6c400 r6:80f15f84 r5:80ff2acc r4:85137000
[443108.035357] [<80849680>] (vchiq_release) from [<80331140>] (__fput+0x90/0x25c)
[443108.035378]  r10:ffffe000 r9:00000000 r8:81bcc3b8 r7:81fb2910 r6:000a001f r5:81e1b088
[443108.035392]  r4:8504ea80
[443108.035416] [<803310b0>] (__fput) from [<80331324>] (____fput+0x18/0x1c)
[443108.035437]  r9:00000000 r8:80f48d1c r7:8addbe00 r6:8addc354 r5:84dff180 r4:8504ea80
[443108.035463] [<8033130c>] (____fput) from [<8014053c>] (task_work_run+0xb4/0xe4)
[443108.035488] [<80140488>] (task_work_run) from [<80123744>] (do_exit+0x468/0xb58)
[443108.035510]  r9:00000000 r8:8615ac48 r7:8242fe94 r6:ffffe000 r5:8615ac00 r4:8addbe00
[443108.035532] [<801232dc>] (do_exit) from [<80123ed0>] (do_group_exit+0x50/0xe0)
[443108.035546]  r7:8afd5940
[443108.035571] [<80123e80>] (do_group_exit) from [<80131500>] (get_signal+0x1c0/0xad0)
[443108.035589]  r7:8afd5940 r6:00000008 r5:00000000 r4:00418004
[443108.035613] [<80131340>] (get_signal) from [<8010b7e0>] (do_work_pending+0x30c/0x5a8)
[443108.035634]  r10:80e05008 r9:fffffe30 r8:5ac3c35a r7:8242ff44 r6:76ba0aec r5:8242ffb0
[443108.035648]  r4:ffffe000
[443108.035669] [<8010b4d4>] (do_work_pending) from [<80100074>] (slow_work_pending+0xc/0x20)
[443108.035684] Exception stack(0x8242ffb0 to 0x8242fff8)
[443108.035702] ffa0:                                     00eb43cc 00000080 00000000 00000000
[443108.035723] ffc0: 7616b82c 00000000 761a4000 000000f0 76edc660 7618fa34 00f5aaa0 00f5ab0c
[443108.035742] ffe0: 698fecc8 698fecb8 76ce1908 76ba0aec 60000010 00eb43cc
[443108.035764]  r10:000000f0 r9:8242e000 r8:80100204 r7:000000f0 r6:761a4000 r5:00000000
[443108.035778]  r4:7616b82c
[443108.035804] INFO: task vcgencmd:472 blocked for more than 614 seconds.
[443108.035818]       Tainted: G         C O      5.10.11-v7+ #1399
[443108.035829] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[443108.035842] task:vcgencmd        state:D stack:    0 pid:  472 ppid:     1 flags:0x00000005
[443108.035867] Backtrace:
[443108.035897] [<809e5190>] (__schedule) from [<809e5b60>] (schedule+0x68/0xe4)
[443108.035919]  r10:83a89400 r9:00000002 r8:7fffffff r7:80e05008 r6:83a8b15c r5:8ae4be00
[443108.035932]  r4:ffffe000
[443108.035954] [<809e5af8>] (schedule) from [<809e9fc4>] (schedule_timeout+0x2c8/0x36c)
[443108.035969]  r5:80e05008 r4:83a8b158
[443108.035991] [<809e9cfc>] (schedule_timeout) from [<809e7114>] (wait_for_completion+0xb8/0x138)
[443108.036012]  r9:00000002 r8:7fffffff r7:ffffe000 r6:83a8b15c r5:80e05008 r4:83a8b158
[443108.036034] [<809e705c>] (wait_for_completion) from [<808497a0>] (vchiq_release+0x120/0x2f4)
[443108.036055]  r9:80ff2b28 r8:80e05008 r7:83a8b000 r6:80f15f84 r5:80ff2acc r4:8acc5000
[443108.036077] [<80849680>] (vchiq_release) from [<80331140>] (__fput+0x90/0x25c)
[443108.036099]  r10:ffffe000 r9:00000000 r8:81bcc3b8 r7:81fb2910 r6:000a001f r5:81e1b088
[443108.036112]  r4:82c39cc0
[443108.036135] [<803310b0>] (__fput) from [<80331324>] (____fput+0x18/0x1c)
[443108.036157]  r9:00000000 r8:80f48d1c r7:8ae4be00 r6:8ae4c354 r5:8389c840 r4:82c39cc0
[443108.036180] [<8033130c>] (____fput) from [<8014053c>] (task_work_run+0xb4/0xe4)
[443108.036203] [<80140488>] (task_work_run) from [<80123744>] (do_exit+0x468/0xb58)
[443108.036225]  r9:00000000 r8:8265f448 r7:8ac4de94 r6:ffffe000 r5:8265f400 r4:8ae4be00
[443108.036247] [<801232dc>] (do_exit) from [<80123ed0>] (do_group_exit+0x50/0xe0)
[443108.036261]  r7:862ef8c0
[443108.036284] [<80123e80>] (do_group_exit) from [<80131500>] (get_signal+0x1c0/0xad0)
[443108.036302]  r7:862ef8c0 r6:00000008 r5:00000000 r4:00418004
[443108.036324] [<80131340>] (get_signal) from [<8010b620>] (do_work_pending+0x14c/0x5a8)
[443108.036346]  r10:80e05008 r9:fffffe30 r8:5ac3c35a r7:8ac4df44 r6:76e78518 r5:8ac4dfb0
[443108.036359]  r4:ffffe000
[443108.036379] [<8010b4d4>] (do_work_pending) from [<80100074>] (slow_work_pending+0xc/0x20)
[443108.036393] Exception stack(0x8ac4dfb0 to 0x8ac4dff8)
[443108.036411] dfa0:                                     fffffffc c01cc402 7e800464 00010001
[443108.036431] dfc0: 76f79404 76f79308 7e8004a8 00000036 0001169c 00022634 ffffffff 00000003
[443108.036450] dfe0: 76f79240 7e800454 76f65928 76e7851c 60000010 00000003
[443108.036471]  r10:00000036 r9:8ac4c000 r8:80100204 r7:00000036 r6:7e8004a8 r5:76f79308
[443108.036485]  r4:76f79404

Additional context camera has been running for 5 days without issues before stopping. The raspberry has not been physically touched in the past week.

arkanoid87 commented 3 years ago

after reboot camera works again. Here you find the complete raspinfo output.

6by9 commented 3 years ago

vcos_abort: Halting is the firmware having encountered an unrecoverable situation.

assert( *p==0xa55a5aa5 ) failed; ../../../../../vcfw/rtos/common/rtos_common_malloc.c::rtos_pool_aligned_free line 205 is that a guard word on an allocation wasn't there when the object was freed, most likely meaning something overran the allocation.

Which version of camera module was this? OV5647 v1, IMX219 v2, or IMX477 HQ? What application were you using to access the camera module? raspistill, raspivid, V4L2, picamera, or something else?

arkanoid87 commented 3 years ago

It's IMX219. I'm accessing it with rpicamsrc https://github.com/thaytan/gst-rpicamsrc that's literally a "GStreamer wrapper around the raspivid/raspistill". It is an isolated case, I've more than one pi running with same software & hardware and only one (so far) exposed this. Actually not 100% sure if os libs are same version, but they came from raspberry os. I can check if needed.