awslabs / amazon-kinesis-video-streams-producer-sdk-cpp

Amazon Kinesis Video Streams Producer SDK for C++ is for developers to install and customize for their connected camera and other devices to securely stream video, audio, and time-encoded data to Kinesis Video Streams.
Apache License 2.0
377 stars 335 forks source link

[QUESTION] RTSP stream -> KVS sink not successfully re-connecting on error #802

Closed mattfeury closed 2 years ago

mattfeury commented 2 years ago

Hello! I've got an RTSP stream that I am pumping to KVS sink via a Gstreamer pipeline (this is done via OpenCV videoWriter so it can simultaneously pump to an appsink.) Command is:

gstreamer_command = (
    f'rtspsrc location="{self.rtspUrl}" short-header=TRUE ! rtph264depay ! tee name=t ! '
    # Send to KVS sink for always-on recording
    f'h264parse ! kvssink stream-name="{self.kvs_stream_name}" storage-size=512 log-config=/opt/kvs_log_configuration access-key="{credentials.access_key}" secret-key="{credentials.secret_key}" aws-region="{AWS_REGION}" '
    # Also send to app sink aka openCV so we can also process frames for other purposes
    f't. ! h264parse ! avdec_h264 ! videoconvert ! videoscale ! video/x-raw ! appsink drop=1'
)

self.videoStream = cv2.VideoCapture(gstreamer_command, cv2.CAP_GSTREAMER)

This works for "some amount of time", but due to what i'm assuming is just bandwidth hiccups, at times, the stream will explode with the logging below.

Logging

Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 (python3.9:1): GStreamer-CRITICAL **: 01:40:16.031: gst_event_set_seqnum: assertion 'seqnum != GST_SEQNUM_INVALID' failed
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 WARN [ WARN:0] global /opencv/modules/videoio/src/cap_gstreamer.cpp (1824) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module source reported: Could not open resource for reading and writing.
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 INFO [INFO ] [08-12-2021 00:39:42:114.250 GMT] stopKinesisVideoStreamSync(): Synchronously stopping Kinesis Video Stream 00007ff6f445d938.
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 INFO [INFO ] [08-12-2021 00:39:42:127.837 GMT] getStreamData(): Handle 0 waiting for last persisted ack with ts 16389239791260000
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 INFO [INFO ] [08-12-2021 00:39:42:258.790 GMT] getStreamData(): Indicating an EOS after last persisted ACK is received for stream upload handle 0
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 INFO [INFO ] [08-12-2021 00:39:42:258.823 GMT] postReadCallback(): Reported end-of-stream for stream 21102860042. Upload handle: 0
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 INFO [INFO ] [08-12-2021 00:39:42:258.834 GMT] Sending eos
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 WARN [WARN ] [08-12-2021 00:40:06:152.600 GMT] putKinesisVideoFrame(): Failed to submit frame to Kinesis Video client. status: 0x52000052 decoding timestamp: 0 presentation timestamp: 0
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 WARN [WARN ] [08-12-2021 00:40:06:152.619 GMT] checkIntermittentProducerCallback(): Failed to submit auto eofr with 0x52000052, for stream name: 21102860042
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 ERROR [ERROR] [08-12-2021 00:40:06:152.622 GMT] checkIntermittentProducerCallback(): operation returned status code: 0x52000052
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 ERROR [ERROR] [08-12-2021 00:40:06:152.625 GMT] timerQueueExecutor(): operation returned status code: 0x52000052
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 WARN [WARN ] [08-12-2021 00:40:11:152.645 GMT] putKinesisVideoFrame(): Failed to submit frame to Kinesis Video client. status: 0x52000052 decoding timestamp: 0 presentation timestamp: 0
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 WARN [WARN ] [08-12-2021 00:40:11:152.667 GMT] checkIntermittentProducerCallback(): Failed to submit auto eofr with 0x52000052, for stream name: 21102860042
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 ERROR [ERROR] [08-12-2021 00:40:11:152.669 GMT] checkIntermittentProducerCallback(): operation returned status code: 0x52000052
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 ERROR [ERROR] [08-12-2021 00:40:11:152.671 GMT] timerQueueExecutor(): operation returned status code: 0x52000052
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 WARN [ WARN:0] global /opencv/modules/videoio/src/cap_gstreamer.cpp (914) open OpenCV | GStreamer warning: unable to start pipeline
Dec 7 19:40:16 test-stream-d0c77bff3f-4ee16a10a0 test-stream-d0c77bff3f-4ee16a10a0 WARN [ WARN:0] global /opencv/modules/videoio/src/cap_gstreamer.cpp (501) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

Any design considerations/constraints I have the kvssink built into this image. Though it's been instantiated via opencv, the gstreamer pipeline should be pretty bare bones. Though I don't have restart-on-error explicitly defined, it seems to be defaulted to true, which i expected it should gracefully restart.

Note that the app-sink in this case continues to work even after the kvs-sink throws the above error and stops working, so it seems like that portion of the pipeline properly recovers.

I know this is likely due to bandwidth/network issues, so i'm less concerned with how do i make sure it doesn't happen and moreso how can i make sure i detect this issue / gracefully recover.

mattfeury commented 2 years ago

note that if i run without opencv (and consequently without the "tee" split to appsink), and manually trigger wifi on and off, the stream does not gracefully recover:

gst-launch-1.0 rtspsrc location="rtsp://IP_ADDRESS:554/axis-media/media.amp?videocodec=h264&fps=5" short-header=TRUE ! rtph264depay ! h264parse ! kvssink stream-name="stream-test-name" storage-size=512 replay-duration =0 access-key="KEY" secret-key="SECRET" aws-region="us-east-1"

works fine for some time, then i trigger wifi on/off to simulate a network hiccup and get:

2021-12-08 19:38:29 [140013148813056] DEBUG - postReadCallback(): Wrote 84 bytes to Kinesis Video. Upload stream handle: 0
2021-12-08 19:38:50 [140013148813056] DEBUG - postReadCallback(): Pausing CURL read for upload handle: 0
2021-12-08 19:38:51 [140013148813056] INFO - stopKinesisVideoStreamSync(): Synchronously stopping Kinesis Video Stream 000055baf2700d18.
2021-12-08 19:38:56 [140013140420352] INFO - getStreamData(): Handle 0 waiting for last persisted ack with ts 16389887082840000
2021-12-08 19:38:56 [140013148813056] DEBUG - postReadCallback(): Pausing CURL read for upload handle: 0 waiting for last ack.
2021-12-08 19:38:56 [140013148813056] WARN - curlCompleteSync(): curl perform failed for url https://s-468461da.kinesisvideo.us-east-1.amazonaws.com/putMedia with result Timeout was reached: Operation too slow. Less than 30 bytes/sec transferred the last 30 seconds
2021-12-08 19:39:05 [140013148813056] WARN - curlCompleteSync(): HTTP Error 0 : Response: (null)
Request URL: https://s-468461da.kinesisvideo.us-east-1.amazonaws.com/putMedia
Request Headers:
    Authorization: AWS4-HMAC-SHA256 Credential=ABCD/20211208/us-east-1/kinesisvideo/aws4_request, SignedHeaders=connection;host;transfer-encoding;user-agent;x-amz-date;x-amzn-fragment-acknowledgment-required;x-amzn-fragment-timecode-type;x-amzn-producer-start-timestamp;x-amzn-stream-name, Signature=e698423ffcf87dcbef7a7541609e4909c6c1e01345015ed
2021-12-08 19:39:05 [140013148813056] DEBUG - putStreamCurlHandler(): Network thread for Kinesis Video stream: matt-rtcc-stream-test with upload handle: 0 exited. http status: 0
2021-12-08 19:39:05 [140013148813056] WARN - putStreamCurlHandler(): Stream with streamHandle 94261419707672 uploadHandle 0 has exited without triggering end-of-stream. Service call result: 599
2021-12-08 19:39:05 [140013148813056] INFO - kinesisVideoStreamTerminated(): Stream 0x55baf2700d18 terminated upload handle 0 with service call result 599.
2021-12-08 19:39:05 [140013148813056] DEBUG - stepStateMachine(): State Machine - Current state: 0x0000000000000100, Next state: 0x0000000000000200
2021-12-08 19:39:05 [140013148813056] WARNING: from element /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0: Could not read from resource.
Additional debug info:
gstrtspsrc.c(5543): gst_rtspsrc_loop_interleaved (): /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0:
The server closed the connection.

i would expect the pipeline to recover. is this not the case? i'm not sure if these are two different things i'm describing, but my overall goal is to create a pipeline that will withstand poor connections and automatically reconnect as it's able.

note that i tried the addition of the following flags here to no avail: framerate=5 connection-staleness=500 replay-duration=0 restart-on-error=true

niyatim23 commented 2 years ago

Hi @mattfeury, the pipeline does not recover because it receives an EOS which is not treated as an error. It closes the stream and then later it errors out with 0x52000052 which is STATUS_STREAM_HAS_BEEN_STOPPED. kvssink is behaving as intended as per the logs shared

Closing assuming answered, feel free to reopen / ask further questions if needed

mattfeury commented 2 years ago

understood. we are now detecting EOS and properly restarting the pipeline as a whole. this was confusing for us because appsink pipelines continue to work even after receiving an EOS, so i wonder if there is opportunity to more gracefully recover to align with how other sinks function. but fine to keep this closed