mmal_vc_port_disable() stuck in mmal_vc_sendwait_message()

Is this the right place for my bug report? This repository contains the Raspberry Pi userland code. If you believe that the issue you are seeing is in the userland libraries, this is the right place. If not, we have other repositories for the GPU firmware at github.com/raspberrypi/firmware and linux kernel at github.com/raspberrypi/linux. If you have problems with the Raspbian distribution packages, report them in the github.com/RPi-Distro/repo. If you simply have a question, then the Raspberry Pi forums are the best place to ask it.

Describe the bug I was working with the RPi camera using picamera v1.13 and found the python doesn't return from stop_recording() sometimes. It is blocked in mmal_vc_sendwait_message() from mmal_port_disable(). So I tried to reproduce with the default image and raspivid, and found it is also reproducible.

To reproduce Run command while true; do raspivid -w 640 -h 480 -b 450000 -fps 24 -t 5000 -v -o tcp://x.x.x.x:yyyy; sleep 3; done, then just leave it there for a while (usually it takes hours to reproduce).

Expected behaviour mmal_port_disable() to return.

Actual behaviour mmal_port_disable() stuck in mmal_vc_sendwait_message(). If I stop raspivid (Ctrl+C) when the issue occurs and rerun it, it will be blocked in the first call to mmal_vc_sendwait_message(). A reboot is required to recover.

System Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

Which model of Raspberry Pi? Pi Zero v1.3 and Pi Camera v2.1
Which OS and version (cat /etc/rpi-issue)? Raspberry Pi reference 2020-05-27 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 825107f04027269db77426046f5085475b1a22f, stage4
Which firmware version (vcgencmd version)? Apr 15 2020 11:44:24 Copyright (c) 2012 Broadcom version 82f9bb929ce2186eb1824178c1ae82902ad6275c (clean) (release) (start_x)
Which kernel version (uname -a)? Linux raspberrypi 4.19.118+ #1311 Mon Apr 27 14:16:15 BST 2020 armv6l GNU/Linux

Logs

 mmal: mmal_port_disconnect: vc.ril.camera:out:0(OPQV)(0xaec1b0)
 mmalipc: mmal_vc_sendwait_message: wait 0xb6efd3c4, reply to 0xbebb5538
 mmalipc: mmal_vc_vchiq_callback: reason 2
 mmalipc: mmal_vc_vchiq_callback: waking up waiter at 0xb6efd3c4
 mmalipc: mmal_vc_vchiq_callback: copying payload @0xbebb5538 to 0xb4b08718 len 28
 mmalipc: mmal_vc_sendwait_message: got reply (len 28/28)
 mmalipc: release_waiter: at 0xb6efd3c4
 mmal: mmal_connection_destroy: 0xb03490, vc.ril.camera:out:1/vc.ril.video_encode:in:0
 mmal: mmal_connection_disable: 0xb03490, vc.ril.camera:out:1/vc.ril.video_encode:in:0
 mmal: mmal_port_disable: vc.ril.camera(3:1) port 0xaec890
 mmalipc: mmal_vc_sendwait_message: wait 0xb6efd3c4, reply to 0xbebb54f0

Additional context I added some logs to vchiq_2835_arm.c in the linux kernel, and found vchiq_doorbell_irq() for BELL0 wasn't triggered.

[17452.138027] vchiq: vchiq_ioctl - instance b52957a8, cmd AWAIT_COMPLETION, arg b6c62d7c
[17452.143248] vchiq: vchiq_ioctl - instance b52957a8, cmd QUEUE_MESSAGE, arg be8e1b64
[17452.143712] vchiq: 0: qm DATA@338f3751,64 (12->90)
[17452.143755] vchiq: Sent: 00000000: 6d 6d 61 6c 0a 00 00 00 e8 05 00 00 dc c1 f1 b6 mmal............
[17452.143768] vchiq: Sent Msg DATA(5) to mmal s:12 d:90 len:100
[17452.143778] vchiq: remote_event_signal BELL2 a=1 e=525227796
[17452.143789] vchiq:   ioctl instance d7879000, cmd QUEUE_MESSAGE -> status 0, 0
#

raspberrypi / userland

mmal_vc_port_disable() stuck in mmal_vc_sendwait_message() #639