search

awslabs / amazon-kinesis-video-streams-producer-sdk-cpp

Amazon Kinesis Video Streams Producer SDK for C++ is for developers to install and customize for their connected camera and other devices to securely stream video, audio, and time-encoded data to Kinesis Video Streams.
Apache License 2.0
373 stars 336 forks source link

[QUESTION] SDK C++ Producer Freezing #657

Closed bml1g12 closed 3 years ago

bml1g12 commented 3 years ago

When running kvs_gstreamer_sample.cpp, it works for several hours but after a seemingly random number of hours (but less than 24 hours) the process seems to freeze with no errors I can see, and no CPU usage.

Do you have any recommendations on how to debug or fix this issue? Thank you!

top shows 0% CPU usage.:

    1 root      20   0   18384      4      4 S   0.0  0.0   0:00.01 kinesis.sh                                                                                                                                                              
   11 root      20   0 1235280  43632  12468 S   0.0  0.3   5:19.93 kvs_gstreamer_s                                                                                                                                                         
  105 root      20   0   18516   3380   3012 S   0.0  0.0   0:00.00 bash                                                                                                                                                                    
  122 root      20   0   38748   3252   2748 R   0.0  0.0   0:00.00 top

The kinesis.sh is just 

AWS_DEFAULT_REGION=$AWS_REGION ./kvs_gstreamer_sample $AWS_VIDEO_STREAM_NAME YOUR_RTSP_STREAM

As far as I can see, there are no errors reported, it is just frozen. I am running inside a docker container (as part of microk8s kubernetes deployment). Restarting the container restores the stream.

Logging

Below is an excerpt of last few hundred lines of the log:

[DEBUG] [09-02-2021 23:31:54:075.617 GMT] fragmentAckReceivedHandler invoked
[DEBUG] [09-02-2021 23:31:54:075.625 GMT] Reporting fragment ack received. Ack timecode 1612913510215
[DEBUG] [09-02-2021 23:31:54:075.637 GMT] postReadCallback(): Wrote 6033 bytes to Kinesis Video. Upload stream handle: 17
[DEBUG] [09-02-2021 23:31:54:075.681 GMT] postReadCallback(): Pausing CURL read for upload handle: 17
[DEBUG] [09-02-2021 23:31:54:180.258 GMT] Kinesis Video client and stream metrics
        >> Overall storage byte size: 134217728
        >> Available storage byte size: 134122975
        >> Allocated storage byte size: 94753
        >> Total view allocation byte size: 144080
        >> Total streams frame rate (fps): 29
        >> Total streams transfer rate (bps): 460480 (449 Kbps)
        >> Current view duration (ms): 100
        >> Overall view duration (ms): 1068
        >> Current view byte size: 26787
        >> Overall view byte size: 93169
        >> Current frame rate (fps): 29.9425
        >> Current transfer rate (bps): 460480 (449 Kbps)
[DEBUG] [09-02-2021 23:31:55:075.771 GMT] postReadCallback(): Wrote 65524 bytes to Kinesis Video. Upload stream handle: 17
[DEBUG] [09-02-2021 23:31:55:075.989 GMT] postReadCallback(): Wrote 1867 bytes to Kinesis Video. Upload stream handle: 17
[DEBUG] [09-02-2021 23:31:55:076.012 GMT] postReadCallback(): Pausing CURL read for upload handle: 17
[DEBUG] [09-02-2021 23:31:55:247.488 GMT] Kinesis Video client and stream metrics
        >> Overall storage byte size: 134217728
        >> Available storage byte size: 134050187
        >> Allocated storage byte size: 167541
        >> Total view allocation byte size: 144080
        >> Total streams frame rate (fps): 29
        >> Total streams transfer rate (bps): 462136 (451 Kbps)
        >> Current view duration (ms): 167
        >> Overall view duration (ms): 2136
        >> Current view byte size: 30648
        >> Overall view byte size: 164421
        >> Current frame rate (fps): 29.9788
        >> Current transfer rate (bps): 462136 (451 Kbps)
[DEBUG] [09-02-2021 23:31:55:252.917 GMT] postWriteCallback(): Curl post body write function for stream with handle: Facility_BagStorage and upload handle: 17 returned: {"EventType":"RECEIVED","FragmentTimecode":1612913511282,"FragmentNumber":"91343852333181699951138633462681958899716587045"}

[DEBUG] [09-02-2021 23:31:55:253.023 GMT] fragmentAckReceivedHandler invoked
[DEBUG] [09-02-2021 23:31:55:253.044 GMT] Reporting fragment ack received. Ack timecode 1612913511282
[DEBUG] [09-02-2021 23:31:55:253.094 GMT] postReadCallback(): Wrote 30648 bytes to Kinesis Video. Upload stream handle: 17
[DEBUG] [09-02-2021 23:31:55:253.261 GMT] postReadCallback(): Pausing CURL read for upload handle: 17
[DEBUG] [09-02-2021 23:31:55:256.903 GMT] postWriteCallback(): Curl post body write function for stream with handle: Facility_BagStorage and upload handle: 17 returned: {"EventType":"BUFFERING","FragmentTimecode":1612913512350,"FragmentNumber":"91343852333181699956090393619823480358147953908"}

[DEBUG] [09-02-2021 23:31:55:256.943 GMT] fragmentAckReceivedHandler invoked
[DEBUG] [09-02-2021 23:31:55:256.957 GMT] Reporting fragment ack received. Ack timecode 1612913512350
[DEBUG] [09-02-2021 23:31:55:364.821 GMT] postWriteCallback(): Curl post body write function for stream with handle: Facility_BagStorage and upload handle: 17 returned: {"EventType":"PERSISTED","FragmentTimecode":1612913511282,"FragmentNumber":"91343852333181699951138633462681958899716587045"}

[DEBUG] [09-02-2021 23:31:55:364.882 GMT] fragmentAckReceivedHandler invoked
[DEBUG] [09-02-2021 23:31:55:364.895 GMT] Reporting fragment ack received. Ack timecode 1612913511282
[DEBUG] [09-02-2021 23:31:55:364.913 GMT] postReadCallback(): Wrote 5582 bytes to Kinesis Video. Upload stream handle: 17
[DEBUG] [09-02-2021 23:31:55:364.947 GMT] postReadCallback(): Pausing CURL read for upload handle: 17
[DEBUG] [09-02-2021 23:31:55:600.383 GMT] postWriteCallback(): Curl post body write function for stream with handle: Facility_BagStorage and upload handle: 17 returned: {"EventType":"RECEIVED","FragmentTimecode":1612913512350,"FragmentNumber":"91343852333181699956090393619823480358147953908"}

[DEBUG] [09-02-2021 23:31:55:600.431 GMT] fragmentAckReceivedHandler invoked
[DEBUG] [09-02-2021 23:31:55:600.449 GMT] Reporting fragment ack received. Ack timecode 1612913512350
[DEBUG] [09-02-2021 23:31:55:600.471 GMT] postReadCallback(): Wrote 11681 bytes to Kinesis Video. Upload stream handle: 17
[DEBUG] [09-02-2021 23:31:55:600.553 GMT] postReadCallback(): Pausing CURL read for upload handle: 17
[DEBUG] [09-02-2021 23:31:55:604.957 GMT] postWriteCallback(): Curl post body write function for stream with handle: Facility_BagStorage and upload handle: 17 returned: {"EventType":"BUFFERING","FragmentTimecode":1612913513418,"FragmentNumber":"91343852333181699961042153776965001551043981746"}

[DEBUG] [09-02-2021 23:31:55:604.985 GMT] fragmentAckReceivedHandler invoked
[DEBUG] [09-02-2021 23:31:55:604.994 GMT] Reporting fragment ack received. Ack timecode 1612913513418
[DEBUG] [09-02-2021 23:31:55:680.101 GMT] postWriteCallback(): Curl post body write function for stream with handle: Facility_BagStorage and upload handle: 17 returned: {"EventType":"PERSISTED","FragmentTimecode":1612913512350,"FragmentNumber":"91343852333181699956090393619823480358147953908"}

[DEBUG] [09-02-2021 23:31:55:680.192 GMT] fragmentAckReceivedHandler invoked
[DEBUG] [09-02-2021 23:31:55:680.213 GMT] Reporting fragment ack received. Ack timecode 1612913512350
[DEBUG] [09-02-2021 23:31:55:680.237 GMT] postReadCallback(): Wrote 3356 bytes to Kinesis Video. Upload stream handle: 17
[DEBUG] [09-02-2021 23:31:55:680.322 GMT] postReadCallback(): Pausing CURL read for upload handle: 17
[DEBUG] [09-02-2021 23:31:56:681.080 GMT] postReadCallback(): Wrote 22074 bytes to Kinesis Video. Upload stream handle: 17
[DEBUG] [09-02-2021 23:31:56:681.244 GMT] postReadCallback(): Pausing CURL read for upload handle: 17
[DEBUG] [09-02-2021 23:32:17:702.257 GMT] postReadCallback(): Wrote 84 bytes to Kinesis Video. Upload stream handle: 17
[DEBUG] [09-02-2021 23:32:17:702.338 GMT] postReadCallback(): Pausing CURL read for upload handle: 17
[DEBUG] [09-02-2021 23:32:17:876.924 GMT] postWriteCallback(): Curl post body write function for stream with handle: Facility_BagStorage and upload handle: 17 returned: {"EventType":"RECEIVED","FragmentTimecode":1612913513418,"FragmentNumber":"91343852333181699961042153776965001551043981746"}
bml1g12 commented 3 years ago

Ah sorry I realise this issue was misreported, actually it is Timeout was reached: Operation too slow. Less than 30 bytes/sec transferred the last 30 seconds

Error combine with for some reason microk8s not restarting the container. I will close this and re-open with more details if I cannot fix.