search

awslabs / amazon-kinesis-video-streams-producer-c

https://awslabs.github.io/amazon-kinesis-video-streams-producer-c/group__PublicMemberFunctions.html
Apache License 2.0
55 stars 77 forks source link

kvs producer sdk hang in stopstream #194

Closed rdkcteam closed 3 years ago

rdkcteam commented 3 years ago

Media pipeline: H26/AAC frame upload

Logging Stack trace from device shows issue with stopsync api call [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/libthread_db.so.1". __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46 46 ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S: No such file or directory. (gdb) bt

0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46

1 0x76f3b900 in __lll_lock_wait (futex=0x12d0ad8, private=0) at lowlevellock.c:43

2 0x76f38918 in __pthread_mutex_cond_lock (mutex=mutex@entry=0x12d0ad8) at ../nptl/pthread_mutex_lock.c:115

3 0x76f39a54 in __pthread_cond_timedwait (cond=, mutex=0x12d0ad8, abstime=0x7ed1a190) at pthread_cond_timedwait.c:262

4 0x765ee0be in defaultConditionVariableWait () from /usr/lib/libKinesisVideoProducer.so

5 0x765e79c8 in stopStreamSync () from /usr/lib/libKinesisVideoProducer.so

6 0x765db48e in stopKinesisVideoStreamSync () from /usr/lib/libKinesisVideoProducer.so

7 0x765ce57a in com::amazonaws::kinesis::video::KinesisVideoStream::stopSync() () from /usr/lib/libKinesisVideoProducer.so

8 0x0001bdaa in ?? ()

9 0x0001c30e in ?? ()

10 0x000204d6 in ?? ()

11 0x000228ac in ?? ()

12 0x0001609c in main ()

Describe the bug We are seeing frame upload application hangs sometimes in stopsync stream api call and application doesn't recover by itself until restarted.

SDK version number SDK 3.0

To Reproduce It's happening intermittent.

Expected behavior Application shouldn't hang in stop stream api call.

MushMal commented 3 years ago

The stopStreamSync is a synchronous API and it will block until

a) A timeout is reached b) All of the buffer content is uploaded successfully and (in case of the persistence enabled) it receives Persisted ACK for the last fragment.

There could be a number of reasons why your stream is unable to successfully finish stopSync so you need to investigate and look at the logs. However, if you need to modify the default timeout value then you can do so by changing: https://github.com/awslabs/amazon-kinesis-video-streams-pic/blob/master/src/client/include/com/amazonaws/kinesis/video/client/Include.h#L1077

Higher-level SDKs (C/C++ producer) will set it's value to 0 which then will trigger the default to be used which is set to 120 seconds.

rdkcteam commented 3 years ago

We are using default timeout as 30 sec so we expect api to stop stream or timeout at max 30 sec which is not happening intermittent and stack trace shows SDK waiting on some condition for lock release.

If SDK doesn't receive persisted ack while uploading left out buffer then will it keep waiting forever or will it timeout in 30 sec ? We are not seeing timeout happening in such scenarios.

MushMal commented 3 years ago

The SYNC API will timeout after the given period. Please make sure you specify the right timeout in the right time units (100ns). You might want to debug into the issue and see what causes the delay. From the stack trace, you are awaiting on the condition variable and that variable is either triggered when the condition is met (success) or it should timeout.

https://github.com/awslabs/amazon-kinesis-video-streams-pic/blob/master/src/client/src/Stream.c#L586

As you can see from the code, the timeout is specified in pKinesisVideoClient->deviceInfo.clientInfo.stopStreamTimeout

hassanctech commented 3 years ago

@rdkcteam Resolving due to staleness. Please re-open if you still need help or open a new issue if you have a new question.