search

awslabs / amazon-kinesis-video-streams-producer-sdk-cpp

Amazon Kinesis Video Streams Producer SDK for C++ is for developers to install and customize for their connected camera and other devices to securely stream video, audio, and time-encoded data to Kinesis Video Streams.
Apache License 2.0
377 stars 336 forks source link

[QUESTION] Bus Error when attempting to stream #686

Closed cwardcode closed 3 years ago

cwardcode commented 3 years ago

I'm trying to stream anything from a Tegra TK1 SOC but I am having an issue where after a couple of seconds, GStreamer crashes with a bus error.

Commands ran:

Logging From running the rtsp source, I receive:

Setting pipeline to PAUSED ...
log4cplus:ERROR could not open file ./kvs_log_configuration
INFO - createKinesisVideoClient(): Creating Kinesis Video Client
2021-03-16 15:10:36 [3070181392] INFO - heapInitialize(): Initializing native heap with limit size 134217728, spill ratio 0% and flags 0x00000001
2021-03-16 15:10:36 [3070181392] INFO - heapInitialize(): Creating AIV heap.
2021-03-16 15:10:36 [3070181392] INFO - heapInitialize(): Heap is initialized OK
2021-03-16 15:10:36 [3070181392] DEBUG - stepStateMachine(): State Machine - Current state: 0x0000000000000001, Next state: 0x0000000000000002
2021-03-16 15:10:36 [3070181392] DEBUG - getSecurityTokenHandler invoked
2021-03-16 15:10:36 [3070181392] DEBUG - Refreshing credentials. Force refreshing: 0 Now time is: 1615921836421171765 Expiration: 0
2021-03-16 15:10:36 [3070181392] DEBUG - stepStateMachine(): State Machine - Current state: 0x0000000000000002, Next state: 0x0000000000000010
2021-03-16 15:10:36 [3070181392] INFO - createDeviceResultEvent(): Create device result event.
2021-03-16 15:10:36 [3070181392] DEBUG - stepStateMachine(): State Machine - Current state: 0x0000000000000010, Next state: 0x0000000000000040
2021-03-16 15:10:36 [3070181392] DEBUG - clientReadyHandler invoked
2021-03-16 15:10:36 [3070181392] INFO - try creating stream
2021-03-16 15:10:36 [3070181392] INFO - Creating Kinesis Video Stream test-stream
2021-03-16 15:10:36 [3070181392] INFO - createKinesisVideoStream(): Creating Kinesis Video Stream.
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo(): Kinesis Video Stream Info
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Stream name: test-stream
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Streaming type: STREAMING_TYPE_REALTIME
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Content type: video/h264
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Max latency (100ns): 600000000
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Fragment duration (100ns): 20000000
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Key frame fragmentation: Yes
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Use frame timecode: Yes
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Absolute frame timecode: Yes
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Nal adaptation flags: 0
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Average bandwith (bps): 4194304
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Framerate: 25
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Buffer duration (100ns): 1200000000
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Replay duration (100ns): 400000000
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Connection Staleness duration (100ns): 600000000
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Store Pressure Policy: 1
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   View Overflow Policy: 1
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Segment UUID: NULL
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Frame ordering mode: 0
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo(): Track list
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Track id: 1
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Track name: kinesis_video
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Codec id: V_MPEG4/ISO/AVC
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Track type: TRACK_INFO_TYPE_VIDEO
2021-03-16 15:10:36 [3070181392] DEBUG - logStreamInfo():   Track cpd: NULL
2021-03-16 15:10:36 [3070181392] DEBUG - stepStateMachine(): State Machine - Current state: 0x0000000000000001, Next state: 0x0000000000000002
2021-03-16 15:10:36 [3070181392] INFO - writeHeaderCallback(): RequestId: e568c6c5-2edc-49e1-b8eb-6bf2a020d385
2021-03-16 15:10:36 [2872464496] DEBUG - describeStreamCurlHandler(): DescribeStream API response: {"StreamInfo":{"CreationTime":1.61591885358E9,"DataRetentionInHours":1,"DeviceName":"Kinesis_Video_Device","IngestionConfiguration":null,"KmsKeyId":"arn:aws:kms:us-east-1:<AccountId>:alias/aws/kinesisvideo","MediaType":"video/h264","Status":"ACTIVE","StreamARN":"arn:aws:kinesisvideo:us-east-1:<AccountId>:stream/test-stream/1615918853580","StreamName":"test-stream","Version":"zcRXVaHBTeWsr2TiQ3HP"}}
2021-03-16 15:10:36 [2872464496] INFO - describeStreamResultEvent(): Describe stream result event.
2021-03-16 15:10:36 [2872464496] DEBUG - stepStateMachine(): State Machine - Current state: 0x0000000000000002, Next state: 0x0000000000000020
2021-03-16 15:10:36 [2872464496] INFO - writeHeaderCallback(): RequestId: 3381f0af-0caf-441c-9b4b-7935b69ee0ca
2021-03-16 15:10:36 [2862609520] DEBUG - getStreamingEndpointCurlHandler(): GetStreamingEndpoint API response: {"DataEndpoint":"https://s-808fdb44.kinesisvideo.us-east-1.amazonaws.com"}
2021-03-16 15:10:36 [2862609520] INFO - getStreamingEndpointResultEvent(): Get streaming endpoint result event.
2021-03-16 15:10:36 [2862609520] DEBUG - stepStateMachine(): State Machine - Current state: 0x0000000000000020, Next state: 0x0000000000000010
2021-03-16 15:10:36 [2862609520] DEBUG - getStreamingTokenHandler invoked
2021-03-16 15:10:36 [2862609520] DEBUG - Refreshing credentials. Force refreshing: 1 Now time is: 1615921836721269015 Expiration: 18446744073709551615
2021-03-16 15:10:36 [2862609520] INFO - getStreamingTokenResultEvent(): Get streaming token result event.
2021-03-16 15:10:36 [2862609520] DEBUG - stepStateMachine(): State Machine - Current state: 0x0000000000000010, Next state: 0x0000000000000040
2021-03-16 15:10:36 [2862609520] DEBUG - streamReadyHandler invoked
2021-03-16 15:10:36 [2862609520] Stream is ready
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://192.168.1.238/stream1
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (request) SETUP stream 1
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
INFO - kinesisVideoStreamFormatChanged(): Stream format changed.
2021-03-16 15:10:37 [2762994800] DEBUG - Dropping frame with flag: 9280
2021-03-16 15:10:37 [2762994800] DEBUG - stepStateMachine(): State Machine - Current state: 0x0000000000000040, Next state: 0x0000000000000080
2021-03-16 15:10:37 [2762994800] INFO - putStreamResultEvent(): Put stream result event. New upload handle 0
2021-03-16 15:10:37 [2762994800] WARN - notifyDataAvailable(): Failed to un-pause curl with error: 43. Curl object 0xa8d12920
2021-03-16 15:10:37 [2762994800] DEBUG - stepStateMachine(): State Machine - Current state: 0x0000000000000080, Next state: 0x0000000000000100
2021-03-16 15:10:37 [2762994800] INFO - writeHeaderCallback(): RequestId: d0729a04-befb-927e-88b9-9eea3fa60bb4
2021-03-16 15:10:38 [2754606192] DEBUG - postReadCallback(): Pausing CURL read for upload handle: 0
2021-03-16 15:10:38 [2754606192] DEBUG - Kinesis Video client and stream metrics
    >> Overall storage byte size: 134217728
    >> Available storage byte size: 134182190
    >> Allocated storage byte size: 35538
    >> Total view allocation byte size: 144080
    >> Total streams frame rate (fps): 25
    >> Total streams transfer rate (bps): 33554432 (32768 Kbps)
    >> Current view duration (ms): 0
    >> Overall view duration (ms): 0
    >> Current view byte size: 35498
    >> Overall view byte size: 35498
    >> Current frame rate (fps): 25
    >> Current transfer rate (bps): 33554432 (32768 Kbps)
2021-03-16 15:10:38 [2762994800] DEBUG - postReadCallback(): Wrote 41246 bytes to Kinesis Video. Upload stream handle: 0
2021-03-16 15:10:39 [2754606192] DEBUG - postReadCallback(): Pausing CURL read for upload handle: 0
2021-03-16 15:10:39 [2754606192] DEBUG - postWriteCallback(): Curl post body write function for stream with handle: test-stream and upload handle: 0 returned: {"EventType":"BUFFERING","FragmentTimecode":1615921838579,"FragmentNumber":"91343852333181610656047719730439511114815731903"}

2021-03-16 15:10:39 [2754606192] DEBUG - fragmentAckReceivedHandler invoked
2021-03-16 15:10:39 [2754606192] DEBUG - postReadCallback(): Wrote 182 bytes to Kinesis Video. Upload stream handle: 0
2021-03-16 15:10:39 [2754606192] DEBUG - postReadCallback(): Pausing CURL read for upload handle: 0
2021-03-16 15:10:39 [2754606192] DEBUG - Kinesis Video client and stream metrics
    >> Overall storage byte size: 134217728
    >> Available storage byte size: 134121836
    >> Allocated storage byte size: 95892
    >> Total view allocation byte size: 144080
    >> Total streams frame rate (fps): 29
    >> Total streams transfer rate (bps): 31876704 (31129 Kbps)
    >> Current view duration (ms): 400
    >> Overall view duration (ms): 988
    >> Current view byte size: 53224
    >> Overall view byte size: 94391
    >> Current frame rate (fps): 29.8239
    >> Current transfer rate (bps): 31876704 (31129 Kbps)
2021-03-16 15:10:39 [2762994800] DEBUG - postReadCallback(): Wrote 65524 bytes to Kinesis Video. Upload stream handle: 0
2021-03-16 15:10:40 [2754606192] DEBUG - postReadCallback(): Wrote 2057 bytes to Kinesis Video. Upload stream handle: 0
2021-03-16 15:10:40 [2754606192] DEBUG - postReadCallback(): Pausing CURL read for upload handle: 0
2021-03-16 15:10:40 [2754606192] DEBUG - postWriteCallback(): Curl post body write function for stream with handle: test-stream and upload handle: 0 returned: {"EventType":"RECEIVED","FragmentTimecode":1615921838579,"FragmentNumber":"91343852333181610656047719730439511114815731903"}

2021-03-16 15:10:40 [2754606192] DEBUG - fragmentAckReceivedHandler invoked
2021-03-16 15:10:40 [2754606192] DEBUG - postWriteCallback(): Curl post body write function for stream with handle: test-stream and upload handle: 0 returned: {"EventType":"BUFFERING","FragmentTimecode":1615921839567,"FragmentNumber":"91343852333181610660999479887581032497036079835"}

2021-03-16 15:10:40 [2754606192] DEBUG - fragmentAckReceivedHandler invoked
2021-03-16 15:10:40 [2754606192] DEBUG - postWriteCallback(): Curl post body write function for stream with handle: test-stream and upload handle: 0 returned: {"EventType":"PERSISTED","FragmentTimecode":1615921838579,"FragmentNumber":"91343852333181610656047719730439511114815731903"}

Bus error
# echo $?
135

I do notice that in dmesg I get a new entry each time after running for an alignment trap error:

[ 4133.495014] Alignment trap: not handling instruction edc10b04 at [<b608ce1c>]
[ 4133.502492] Unhandled fault: alignment exception (0x811) at 0xabb6eaea

I receive almost identical logs when using videotestsrc as well.

Any design considerations/constraints The stream is currently set at 1080p@30fps.

If I run any of the above commands in an infinite while-loop, it will stream to KVS, however the video feed is very choppy and not suitable for a production scenario.

I'm running a buildroot-based distro that's based on the 2020.02.3 release. This is using the linux kernel pubished by toradex for this TK1 SOC (located here with version Apalis-TK1_LXDE-Image_2.8b6.184-20190401.

If you would not like to open an issue to discuss your solution in open-platform, please email your question to kinesis-video-support@amazon.com

disa6302 commented 3 years ago

@cwardcode ,

The logs do not give information about what could be causing it. Besides, it might also not be related to KVS. To verify this, can you run with fakesink instead of kvssink to see if you notice any crash?

Also, I assume it is a typo here for h264parse in this pipeline: (there is a space) videoconvert ! x264enc bframes=0 key-int-max=45 bitrate=512 ! video/x-h264,profile=baseline,stream-format=avc,alignment=au,width=640,height=480,framerate=30/1 ! _**h264pars e**_ ! kvssink stream-name=test-stream storage-size=512 aws-region="us-east-1" retention-period=1

One more thing you can do is run the pipeline with GST_DEBUG=4 or GST_DEBUG=5 to get information on why the GStreamer pipeline crashes.

MushMal commented 3 years ago

Seems like your device does not support unaligned access. You will need to build PIC with only aligned memory access.

In order to prove this. Can you run your solution on a device that does support unaligned access and ensure you don't get bus exception.

Check out the instructions for aligned mem only. https://github.com/awslabs/amazon-kinesis-video-streams-pic#note-on-alignment

You need to specifyDALIGNED_MEMORY_MODEL=TRUE

cwardcode commented 3 years ago

Hi @disa6302 and @MushMal, thanks for the quick replies!

The extra space in the command I previously posted was just a typo.

I just tested fakesink and it worked as intended. Enabling GST_DEBUG=4 showed the continual flow of information. Here is some of the output.

I would post the output, but I would like to try @MushMal's suggestion first before flooding the comments with the tons of output that come from the DEBUG flag. I'm in the middle of recompiling with the latest kernel to see if there's any improvement there, so I'll try compiling with DALIGNED_MEMORY_MODEL=TRUE right after.

cwardcode commented 3 years ago

@MushMal

You need to specify DALIGNED_MEMORY_MODEL=TRUE

This got it, thanks! I had actually read the readme for the PIC repo the other day but for some reason it never clicked. Thanks again for the quick responses!