Closed paco-spr closed 5 months ago
@paco-spr Thanks for the report, do you have by any chance, any knowledge if this is only happening on 32 bit devices or builds? (I ask because of the call to software division, __udivdi3
)
@paco-spr Thanks for the report, do you have by any chance, any knowledge if this is only happening on 32 bit devices or builds? (I ask because of the call to software division,
__udivdi3
)
Sorry for the delay in getting back to you.
It looks like firebase lists a couple different variants for this issue. The variant containing the __udivdi3 call happens significantly less than this variant:
null pointer dereference: SIGSEGV 0x0000000000000092
#00 pc 0x158fec libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#01 pc 0x158fd4 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#02 pc 0x159444 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#03 pc 0x1596b8 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#04 pc 0x15e2e4 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#05 pc 0x15e578 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#06 pc 0x17fa44 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#07 pc 0x3a6af0 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#08 pc 0x3a53f4 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#09 pc 0x3a427c libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#10 pc 0x3a5990 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#11 pc 0xbb6e8 libc.so (BuildId: 5111da12b6134a422c2730af0cf237cb)
#12 pc 0x54fdc libc.so (BuildId: 5111da12b6134a422c2730af0cf237cb)
However, I've checked both variants, and they both seem to happen on 64 bit devices such as Galaxy A13, Redmi A1+, Galaxy S22 Ultra.
Hi @afalls-twilio , any update around this?
Looking at all our error logs we are seeing around 3k crashes per month around the native voice library.
Below I'm pasting the most common traces that we are seeing:
Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString
null pointer dereference: SIGSEGV 0x0000000000000000
#00 pc 0x3bd8d0 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 753c8c664d7e4c562d64143915c5fd60f776e52e)
_Unwind_GetTextRelBase
Crashed: Thread: SIGSEGV 0x000492efe99abc00
#00 pc 0x85b4e0 libtwilio_voice_android_so.so (_Unwind_GetTextRelBase) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#01 pc 0x85b4c8 libtwilio_voice_android_so.so (_Unwind_GetTextRelBase) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#02 pc 0x85c8f4 libtwilio_voice_android_so.so (_Unwind_Backtrace) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#03 pc 0x39e014 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
Java_com_twilio_voice_Call_nativeDisconnect
null pointer dereference: SIGSEGV 0x0000000000000000
#00 pc 0x38100c libtwilio_voice_android_so.so (BuildId: 753c8c664d7e4c562d64143915c5fd60f776e52e)
#01 pc 0x389a04 libtwilio_voice_android_so.so (Java_com_twilio_voice_Call_nativeDisconnect) (BuildId: 753c8c664d7e4c562d64143915c5fd60f776e52e)
Any recommendation or tip as to how to avoid them or fix them (on our end if possible) would be much appreciated it.
jamesbluecrow what is your build setup?
or how is the application installed on the user's system? is it a preloaded app, or are their other versions twilio voice somehow on the same application folder?
The other thing, have you been able to reproduce this with the quickstart?
Is your application exposing this C API (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString)? It is a JNI binding method that the JVM will bind to the java class com.twilio.voice.JniUtils which does not exist as of the 6.3.2 (the stated release) release. So unless it was exposed some other way, its not exposed to the JAVA layer and is not linked/bound to any java class... its dead legacy code (not called by anyone).
jamesbluecrow what is your build setup?
or how is the application installed on the user's system? is it a preloaded app, or are their other versions twilio voice somehow on the same application folder?
The other thing, have you been able to reproduce this with the quickstart?
Is your application exposing this C API (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString)? It is a JNI binding method that the JVM will bind to the java class com.twilio.voice.JniUtils which does not exist as of the 6.3.2 (the stated release) release. So unless it was exposed some other way, its not exposed to the JAVA layer and is not linked/bound to any java class... its dead legacy code (not called by anyone).
The users are installing the app thorough google play.
It is not a preloaded app and there is no other versions of twilio voice:
We have this in our gradle (copied from the quickstart):
splits {
abi {
isEnable = true
reset()
// Specify each architecture currently supported by the Twilio Voice SDK
include("armeabi-v7a", "arm64-v8a", "x86", "x86_64")
// Specify that we do not want an additional universal SDK
isUniversalApk = false
}
}
dependencies {
// com.twilio:voice-android:6.4.0
implementation(libs.twilio.voice.android)
// com.twilio:audioswitch:1.1.8
implementation(libs.twilio.audio.switch)
}
I haven't been able to reproduce this, neither with the quickstart, or in our app in debug or even in our release, but we are definitely seeing some users affected by this, we have around 3k crashes monthly coming from the native Twilio SDK.
We only started noticing this when we enabled crashlytics ndk reports so they were likely happening before but we hadn't noticed.
We are seeing this happening in different versions of Android 8, 9, 10, 11, 12, 13 but it id not happening to all users on those versions.
We also see a variety of manufacturers & models (Samsung, Xiaomi, Motorola, etc) so it is not related to a specific manufacturer/model either.
Proguard is just configured as as the quickstart project
# Twilio Programmable Voice
-keep class com.twilio.** { *; }
-keep class tvo.webrtc.** { *; }
-dontwarn tvo.webrtc.**
-keep class com.twilio.voice.** { *; }
-keepattributes InnerClasses
# needed with AGP 8.x
-dontwarn android.content.pm.PackageManager$ApplicationInfoFlags
-dontwarn android.content.pm.PackageManager$PackageInfoFlags
...
and the most common crashes come from one of these 3 functions calls:
Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString
null pointer dereference: SIGSEGV 0x0000000000000048
#00 pc 0xae75a libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#01 pc 0x5aa12e libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#02 pc 0x5aa13a libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#03 pc 0x5aa12e libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#04 pc 0x690782 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#05 pc 0x9fd53 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#06 pc 0xcc4be [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#07 pc 0x690782 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#08 pc 0x663c5a libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#09 pc 0x69075a libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#10 pc 0xaea73 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#11 pc 0x5aa283 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#12 pc 0x5aa2fa libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#13 pc 0x5aa283 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#14 pc 0xcc4be [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#15 pc 0x663c46 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#16 pc 0x8b105 libtwilio_voice_android_so.so (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#17 pc 0xaec97 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#18 pc 0xa29dd libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#19 pc 0x504087 libtwilio_voice_android_so.so (std::__ndk1::__shared_weak_count::__add_weak()) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#20 pc 0x9824d libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#21 pc 0xb0e67 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#22 pc 0x5aac86 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#23 pc 0x5aac90 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#24 pc 0x5aac86 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#25 pc 0xcc4be [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#26 pc 0xb1051 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#27 pc 0x504087 libtwilio_voice_android_so.so (std::__ndk1::__shared_weak_count::__add_weak()) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#28 pc 0x98561 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#29 pc 0xc27ab libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#30 pc 0x514525 libtwilio_voice_android_so.so (__gnu_ldivmod_helper) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#31 pc 0x22a223 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#32 pc 0x2295c7 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#33 pc 0x229117 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#34 pc 0x229117 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#35 pc 0x5a47eb libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#36 pc 0x228bcb libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#37 pc 0x5cc068 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#38 pc 0x5cbf4a libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#39 pc 0x229879 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#40 pc 0xade0b [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#41 pc 0xadde1 [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#42 pc 0x63e73 [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#43 pc 0xadde1 [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#44 pc 0x229851 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
Java_com_twilio_voice_Call_nativeDisconnect
null pointer dereference: SIGSEGV 0x0000000000000009
#00 pc 0x38100c libtwilio_voice_android_so.so (BuildId: 753c8c664d7e4c562d64143915c5fd60f776e52e)
#01 pc 0x389a04 libtwilio_voice_android_so.so (Java_com_twilio_voice_Call_nativeDisconnect) (BuildId: 753c8c664d7e4c562d64143915c5fd60f776e52e)
I can see in the logs that it is usually after a call has ended (but I can't confirm that is the case 100% of the time).
We are honestly a bit at a lost with this and not sure how to proceed/fix it.
yeah you and me both...
what is weird is that the method Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString isn't even exposed, its dead code. I know you said that they were using 6.3.2, but is it possible that these crashes happening on old versions of our SDK?
I'll throw in a ticket on our end to remove that dead function.. as for clients in the field... I have no idea at this point in time.. I have not been able to repo on our end.
Definitely happening in newer versions. We have been using 6.4.0 for the last couple of versions of our app and still see crashes in Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString
.
That code must somehow be used as we obviously don't interact with the native code itself. Maybe it is some kind of internal thread of the native sdk calling itself?
The second and third biggest sources of crashes coming from the native sdk are in case they are related:
_Unwind_GetTextRelBase
and Java_com_twilio_voice_Call_nativeDisconnect
.
We will keep an eye on when a new version is published and we will see what happens.
Ps. Many thanks for the effort you are putting to look into this 🙏 .
Ps2. Proportionally with the amount of use we do of the SDK the crashes are small, but at the moment the Twilio SDK is the biggest source of crashes in our app which is something we need to address obviously
Hello, just my two cents, We have had the same issue with the same stacktrace and similar frequence (1-2% of users) : Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString It was occurring on calls lasting more than 10 minutes, but it was fixed in Twilio SDK 6.3.2. So it's not exactly the same issue but in case it could help...
We also reduced the number of crashes by making sure that all calls to Twilio SDK are wrapped within the same thread.
@YvesDelerm yes, all the Twilio api calls must come from the same thread, its not thread safe.. As for Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString, we recently removed it entirely (not yet released), its was not being called internally either, it was a vestige from the past that was just dead. It is a complete mystery how it is ever being invoked....
@YvesDelerm yes, all the Twilio api calls must come from the same thread, its not thread safe.. As for Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString, we recently removed it entirely (not yet released), its was not being called internally either, it was a vestige from the past that was just dead. It is a complete mystery how it is ever being invoked....
Ok cool.
Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString
is available, we will keep an eye on the crashes after we update to it._Unwind_GetTextRelBase
and Java_com_twilio_voice_Call_nativeDisconnect
? They happen way less often, but we we still see around 200 crashes per month for each of them.As for _Unwind_GetTextRelBase
, thats a libc++ method that gets called when a c++ exception is triggered, again its very weird that there was crash there and for Java_com_twilio_voice_Call_nativeDisconnect
while I was unable to repo it, that method is what is gets called when the SDK is told to disconnect and in the past, I have seen crashes there when users attempt to call it or another sdk method from a different thread than the thread they have made all other SDK calls from. This is especially true when users call the stats functions from a different thread than the thread they are calling disconnect from. Another common error is that they make calls SDK calls from a thread that does not have a handler... ie if you are not making SDK calls from the Main thread, make sure the thread you are invoking the sdk from is a HandlerThread, otherwise the call backs will be routed to the main thread (and any subsequent SDK method invocations from those callbacks/notify methods will have been made from a different thread).
As for
_Unwind_GetTextRelBase
, thats a libc++ method that gets called when a c++ exception is triggered, again its very weird that there was crash there and forJava_com_twilio_voice_Call_nativeDisconnect
while I was unable to repo it, that method is what is gets called when the SDK is told to disconnect and in the past, I have seen crashes there when users attempt to call it or another sdk method from a different thread than the thread they have made all other SDK calls from. This is especially true when users call the stats functions from a different thread than the thread they are calling disconnect from. Another common error is that they make calls SDK calls from a thread that does not have a handler... ie if you are not making SDK calls from the Main thread, make sure the thread you are invoking the sdk from is a HandlerThread, otherwise the call backs will be routed to the main thread (and any subsequent SDK method invocations from those callbacks/notify methods will have been made from a different thread).
Perfect, we are revisiting how we call the SDK to make sure that we always use the Main
thread.
Thanks for the response and the tips 🙏
FYI, the release made yesterday (6.4.1) contains the removal of the unused method Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString
Great, thanks for letting me know.
We will update to the latest version and will let you know in a week or so when we release if we stop seeing crashes.
Hi @jamesbluecrow - did you get a chance to validate Voice Android 6.4.1 release?
Hey @kbagchiGWC @afalls-twilio!
Yes, I can confirm that in com.twilio:voice-android:6.4.1
the crash seems gone to be gone, I haven't seen a single instance of it and we have a fair amount of users running the latest version.
I think this issue can now be closed.
Thank you for your help folks 🙏
@jamesbluecrow Thanks for validation the behavior. I will close the ticket as completed.
Description
Voice NDK crashes com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString
Steps to Reproduce
Unknown, these issues are being surfaced to us via crashlytics.
Code
Proguard rules being used:
Expected Behavior
The app does not crash
Actual Behavior
The app crashes
Reproduces How Often
About 1-2% of our users have experienced one of these 2 crashes.
Twilio Call SID(s)
Unknown.
Logs
Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString
Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString
Versions
Kotlin: 1.9.0 Android gradle plugin: 8.1.1 com.twilio:audioswitch: 1.1.8
Voice Android SDK
com.twilio:voice-android:6.3.2
OS Version
Various but primarily: 14, 13, 12, 10
Device Model
Various: Samsung, Xiaomi, Google, etc.