search

ambiot / ambpro2_arduino

AmebaPro2 Arduino third-party package SDK
https://www.amebaiot.com/en/ameba-arduino-summary/
MIT License
45 stars 20 forks source link

RSTP Crashes After Several Hours? #237

Open collina opened 4 months ago

collina commented 4 months ago

Boards

AMB82-Mini

External Hardware

None.

Hardware Configuration

None.

Version

latest dev (checkout manually)

IDE Name

Arduino 2.3.1

Operating System

-

Auto Flash Mode

Disable

Erase All Flash Memory (16MB)

Disable

Standard Lib

Arduino_STD_PRINTF

Upload Speed

2000000

Description

After a few hours I've noticed that the RTSP service on the default video example will drop, and clients will no longer be able to connect. The AMB82-Mini is still connected to the network, and will respond to a ping, but RTSP no longer serves up the stream. Given the lack of methods for querying server status, it's hard to diagnose what's going on.

I wouldn't mind even an occasional crash, since it'd be sufficient to patch over a hung RTSP process by querying whether its served a connection recently. But I can't find any method for learning more about current clients or activity.

Tried on master and dev, still have issues. Any recommendations for how to diagnose?

Sketch

(Default SingleVideo)

Error/Debug Message

None

Reproduce remarks

No response

I have checked online documentation, FAQ, GitHub Wiki and existing/closed issues.

github-actions[bot] commented 4 months ago

Hello, hope this message finds you well. Congrats to your first Issue! We will review it as soon as possiable. Feel free to have a look at https://www.amebaiot.com/en/ameba-arduino-summary/ for more information

Kyderio commented 3 months ago

Hi, I am trying to replicate this issue. Can I double confirm if there is no network drop in between the period you are streaming the RTSP video?

Kyderio commented 3 months ago

Hi, I have run this for several hours and seems like there is no crash happening. Can you try to connect to a 5GHz Wifi Network and try again? Let me know if you have found a reliable way to replicate this issue. Thanks!

collina commented 3 months ago

Thanks! So I also flashed the 'realtek_amebapro2_v0_example' example and got similar behavior. The client will run for anywhere between hours and days, and then fail.

It seems, prior to crash, like I get the ENC Queue full message repeatedly, and then it tries to tear down the connection but fails. At that point it loops on the empty request. From there, the RTSP is basically being monopolized by the broken client connection.

I'm guessing that the client is falling and that the AMB libraries don't handle that as gracefully as I would have hoped.

I haven't yet reproduced how to cause this.

Any ideas on how to get the AMB to notice it's stuck and free up the RTSP connection? I'd like not to be totally dependent on the client for this to work right.

Appreciate any advice.

[VID Wrn]
 CH 0 MMF ENC Queue full 

[RTP Err][H264] RTP send fail 
[Local] REQUEST_TEARDOWN

[RTSP Wrn]send TEARDOWN response failed![Local] Socket closed

[VID Wrn]
 CH 0 MMF ENC Queue full 

rtp stopped (TCP)

[Local] Connected by 10.0.1.10
[Local] Enter rtsp request_loop

[RTSP Err]Receive request len <=0, close socket
[Local] Socket closed
Kyderio commented 3 months ago

Hi, can I check what is your set up?

Are you running this Default SingleVideo example and then viewing this on VLC?

Thank you.

collina commented 3 months ago

Thanks for asking, I've tried with SingleVideo as well as other examples, and then viewing on ffmpeg. It's not the viewer.

I think what happens is that if there's a network failure, e.g. the wifi loses connection, either the rtsp or the AV background fills up a queue and crashes, even if the connection loss was only temporary and the wifi didn't disassociate. So, I can trigger this error by briefly putting the AMB behind a brick wall where there's going to start to be serious packet loss. Once it gets into that state, it doesn't recover.

That'd be unfortunately but easy to work around, exempt I can't figure out a way of detecting with this failure programmatically.

Any ideas?

Kyderio commented 3 months ago

Hi, maybe you can forcefully reset the connection periodically, or add a watchdog to check for some condition where there is serious packet loss to reset the connection.

collina commented 3 months ago

Hi, maybe you can forcefully reset the connection periodically, or add a watchdog to check for some condition where there is serious packet loss to reset the connection.

I actually tried that, using NVIC_SystemReset when there isn't connectivity. Unfortunately, sometimes that worked, but other times NVIC_SystemReset hanged. Is there a better way to reset the device (which is probably a cleaner way of dealing with a failure?)?

github-actions[bot] commented 2 months ago

This issue is stale because it has been open for 14 days with no activity.

Kyderio commented 2 months ago

Hi, sorry for the late reply, there is an API called sys_reset(). What you need to do is to #include "sys_api.h" in your Arduino sketch and then use the API sys_reset(); Can you give it a try and see if the reset is much smoother with no hang? Thank you.

Kyderio commented 2 months ago

You can also refer to using Watchdog to do a reset. https://www.amebaiot.com/en/amebapro2-arduino-wdt/