Server is crashing while opens on web browser.

[MohitMaliFtechiz]

We have refactored the kiwix-android code to support the latest java-libkiwix wrapper. All the other functionality are working fine with the changes of the newest wrapper, but there is a problem with the Kiwix server. The server starts normally from the application, there is no error but when we try to open the URL(server) in the web browser then it is crashing with these error logs.

Logs:

pid: 16875, tid: 17087, name: MHD-single  >>> org.kiwix.kiwixmobile <<<
#04 pc 0000000000091b6c  /data/app/~~agmIistDXIVViCCvMzpIZw==/org.kiwix.kiwixmobile-_OPuxj8AMgZ6OUuAVYejaA==/base.apk!libc++_shared.so (std::__ndk1::mutex::lock()+8) (BuildId: ece72a2ebc3774a1be9fd21271258acd3bcdfaa7)
#05 pc 0000000000176b84  /data/app/~~agmIistDXIVViCCvMzpIZw==/org.kiwix.kiwixmobile-_OPuxj8AMgZ6OUuAVYejaA==/base.apk!libkiwix.so (kiwix::Library::getRevision() const+16)
#06 pc 00000000001c3ebc  /data/app/~~agmIistDXIVViCCvMzpIZw==/org.kiwix.kiwixmobile-_OPuxj8AMgZ6OUuAVYejaA==/base.apk!libkiwix.so (kiwix::InternalServer::getLibraryId() const+48)
#07 pc 00000000001c337c  /data/app/~~agmIistDXIVViCCvMzpIZw==/org.kiwix.kiwixmobile-_OPuxj8AMgZ6OUuAVYejaA==/base.apk!libkiwix.so (kiwix::InternalServer::handle_request(kiwix::RequestContext const&)+60)
#08 pc 00000000001c303c  /data/app/~~agmIistDXIVViCCvMzpIZw==/org.kiwix.kiwixmobile-_OPuxj8AMgZ6OUuAVYejaA==/base.apk!libkiwix.so (kiwix::InternalServer::handlerCallback(MHD_Connection*, char const*, char const*, char const*, char const*, unsigned long*, void**)+796)
#09 pc 00000000004e8740  /data/app/~~agmIistDXIVViCCvMzpIZw==/org.kiwix.kiwixmobile-_OPuxj8AMgZ6OUuAVYejaA==/base.apk!libkiwix.so

Version: latest main branch

[kelson42]

@MohitMaliFtechiz How is that different from what you report in https://github.com/kiwix/kiwix-android/pull/3218 ?

[MohitMaliFtechiz]

@kelson42, in https://github.com/kiwix/kiwix-android/pull/3218 we are experiencing crashes when trying to implement the checkbox functionality. This functionality is meant to stop the server and start it again if we select any zim file, and this issue occurred sometime not every time also without the checkbox functionality it is working fine. Currently, the server is not starting, and it crashes every time we try to open it on a web browser. The error logs is also different which is reported in the https://github.com/kiwix/kiwix-android/pull/3218.

[mgautierfr]

@MohitMaliFtechiz Why do we have this line : https://github.com/kiwix/java-libkiwix/blob/main/lib/src/main/cpp/CMakeLists.txt#L82

From this documentation we should not use static libc++ as we now use several dynamic libraries. And if we do so, we must be sure to include the library as it is not standard.

[MohitMaliFtechiz]

@mgautierfr, This was introduced when we initially refactoring the wrapper that time there is no multiple dynamic libraries so that time we introduced but over the period we have improved many things in our wrapper code and this code is missed to refactor. Instead of the llvm-libc++_static we should use the llvm-libc++_shared as described in the documentation. I have tried to test after changing it but the error is the same as the logs mentioned on the ticket.

Apart from this, we have one more c++_shared.so file generated in the aar file as shown in the below image, so should we load this library as well, as we are loading other .so files in JNIKiwix class?

[mgautierfr]

Apart from this, we have one more c++_shared.so file generated in the aar file as shown in the below image, so should we load this library as well, as we are loading other .so files in JNIKiwix class?

I think so. At least, it should not be a problem to load it even if not needed.

04 pc 0000000000091b6c /data/app/~~agmIistDXIVViCCvMzpIZw==/org.kiwix.kiwixmobile-_OPuxj8AMgZ6OUuAVYejaA==/base.apk!libc++_shared.so (std::__ndk1::mutex::lock()+8) (BuildId: ece72a2ebc3774a1be9fd21271258acd3bcdfaa7)

What is surprising here is that the error came from libc++_shared.so but we were using llvm-libc++_static. I think there is a mess somewhere in the linked libraries.

[MohitMaliFtechiz]

I think so. At least, it should not be a problem to load it even if not needed.

I have tried loading this lib since it is showing an error on libc++_shared.so, but the error is the same.

I think there is a mess somewhere in the linked libraries.

Might be possible, since only this part is crashing otherwise there is no error while starting the server.

[mgautierfr]

Which version of android it is ?

[MohitMaliFtechiz]

@mgautierfr, I am currently testing on Android 12.

[mgautierfr]

Which version of NDK and SDK it is ?

[MohitMaliFtechiz]

Android SDK is 31(Android 12) and we are compiling the project with SDK version 33 with gradle 8.0 whose default NDK version is 25.1.8937393 https://developer.android.com/studio/projects/install-ndk#default-ndk-per-agp.

[kelson42]

@MohitMaliFtechiz What is the status here, this bug is top prio.

[MohitMaliFtechiz]

@kelson42, this bug still exist @mgautierfr tyring to debug the problem.

[MohitMaliFtechiz]

@mgautierfr here is the Arm64-v8a apk for testing why server is crashing on this arch https://drive.google.com/file/d/1lUXKU_yrW1A5BpJO29q30gWJC3VCeaQb/view?usp=sharing

UniversalApk: https://drive.google.com/file/d/1O2iooYteZfZAHVoGujPGxdCBqnwXJ_Ab/view?usp=sharing

[MohitMaliFtechiz]

@mgautierfr, thank you for providing additional steps to debug the actual problem. I tried building the APK using the steps mentioned in the Stack Overflow link (https://stackoverflow.com/questions/40355200/how-to-avoid-stripping-for-native-code-symbols-for-android-app), but the error remains the same. This indicates that the libraries are not being stripped in the APK since other library functions are working correctly.

However, there are some points to consider:

• When running on the x86_64 emulator, the server loaded homepage for the first time when I tried on this arch. However, when attempting to open the server again on the same emulator, it does not open, and the page keeps loading endlessly. There is no crash, but it does not work as expected.
• On the Redmi Note 9 (arm64-v8a), the URL is loaded, but the app crashes after that.

Upon checking the full logcat, I found an extra log from the lib side that indicates the error is happening because we are trying to call pthread_mutex_lock on a destroyed mutex.

The server runed successfully once on the x86_64 emulator might be the mutex is not destroyed due to some condition, although I'm not entirely sure about this.

Here are the new error logs:

1728-1992  libc                    org.kiwix.kiwixmobile                A FORTIFY: pthread_mutex_lock called on a destroyed mutex (0xb4000076f11b9f58)
libc                    org.kiwix.kiwixmobile                A  Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 1992 (MHD-single), pid 1728 (wix.kiwixmobile) Cmdline: org.kiwix.kiwixmobile
A  pid: 1728, tid: 1992, name: MHD-single  >>> org.kiwix.kiwixmobile <<<
DEBUG                   pid-2210                             A        #04 pc 0000000000091b6c  /data/app/~~D96Q3YUrUh_Vg29ncY4cpQ==/org.kiwix.kiwixmobile-A_twmbyoQGkxyVP8WNvdzw==/base.apk!libc++_shared.so (std::__ndk1::mutex::lock()+8) (BuildId: ece72a2ebc3774a1be9fd21271258acd3bcdfaa7)
DEBUG                   pid-2210                             A        #05 pc 0000000000176b84  /data/app/~~D96Q3YUrUh_Vg29ncY4cpQ==/org.kiwix.kiwixmobile-A_twmbyoQGkxyVP8WNvdzw==/base.apk!libkiwix.so (kiwix::Library::getRevision() const+16)
DEBUG                   pid-2210                             A        #06 pc 00000000001c3ebc  /data/app/~~D96Q3YUrUh_Vg29ncY4cpQ==/org.kiwix.kiwixmobile-A_twmbyoQGkxyVP8WNvdzw==/base.apk!libkiwix.so (kiwix::InternalServer::getLibraryId() const+48)
DEBUG                   pid-2210                             A        #07 pc 00000000001c337c  /data/app/~~D96Q3YUrUh_Vg29ncY4cpQ==/org.kiwix.kiwixmobile-A_twmbyoQGkxyVP8WNvdzw==/base.apk!libkiwix.so (kiwix::InternalServer::handle_request(kiwix::RequestContext const&)+60)
DEBUG                   pid-2210                             A        #08 pc 00000000001c303c  /data/app/~~D96Q3YUrUh_Vg29ncY4cpQ==/org.kiwix.kiwixmobile-A_twmbyoQGkxyVP8WNvdzw==/base.apk!libkiwix.so (kiwix::InternalServer::handlerCallback(MHD_Connection*, char const*, char const*, char const*, char const*, unsigned long*, void**)+796)
DEBUG                   pid-2210                             A        #09 pc 00000000004e8740  /data/app/~~D96Q3YUrUh_Vg29ncY4cpQ==/org.kiwix.kiwixmobile-A_twmbyoQGkxyVP8WNvdzw==/base.apk!libkiwix.so
---------------------------- PROCESS ENDED (1728) for package org.kiwix.kiwixmobile ----------------------------

Is it possible with java-libkiwix to test at least home page of the server?

[mgautierfr]

When running on the x86_64 emulator, the server loaded homepage for the first time when I tried on this arch. However, when attempting to open the server again on the same emulator, it does not open, and the page keeps loading endlessly. There is no crash, but it does not work as expected.

Interesting, If the page keeps loading endlessly, it may be because the mutex is not unlocked. It may be possible if a thread crash while having locked the mutex and so it doesn't unlock the mutex. Do you have a trace of a "crash" in your log ?

Upon checking the full logcat, I found an extra log from the lib side that indicates the error is happening because we are trying to call pthread_mutex_lock on a destroyed mutex.

Do you have that log ?

[MohitMaliFtechiz]

@mgautierfr There is not much information available about the crash because the crash happened in the native lib so Android does not provide full error logs about the crash but it provides the main error and its possible location in the native lib. This is full logs after starting the server.

V  ZIM PATH : /storage/sdcard0/Kiwix/testzim.zim
V  ZIM PATH : /storage/sdcard0/Kiwix/alpinelinux_en_all_maxi_2023-01.zim
V  ZIM PATH : /storage/sdcard0/Kiwix/alpinelinux_en_all_nopic_2023-01.zim
D  onSuccess:  192.168.1.115
D  Server status true
D  Compat change id reported: 147798919; UID 10258; state: ENABLED
I  releaseBufferCallbackThunk bufferId:111351322116106 framenumber:62 blastBufferQueue is dead
D  onWindowFocusChanged hasWindowFocus true
I  Explicit concurrent copying GC freed 7832(389KB) AllocSpace objects, 0(0B) LOS objects, 49% free, 8686KB/16MB, paused 54us,52us total 39.312ms
E  getRecentTasks: taskId=532   userId=0   baseIntent=Intent { act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10200000 cmp=org.kiwix.kiwixmobile/.main.KiwixMainActivity }
D  onWindowFocusChanged hasWindowFocus false
E  getRecentTasks: taskId=532   userId=0   baseIntent=Intent { act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10200000 cmp=org.kiwix.kiwixmobile/.main.KiwixMainActivity }
I  Explicit concurrent copying GC freed 8598(489KB) AllocSpace objects, 3(48KB) LOS objects, 49% free, 8692KB/16MB, paused 83us,55us total 57.818ms
I  [ViewRootImpl[KiwixMainActivity]#0(BLAST Consumer)0](id:654600000000,api:1,p:25926,c:25926) disconnect: api 1
I  [ViewRootImpl[KiwixMainActivity]#0] destructor()
I  [ViewRootImpl[KiwixMainActivity]#0(BLAST Consumer)0](id:654600000000,api:0,p:-1,c:25926) disconnect
--------- beginning of crash
A  FORTIFY: pthread_mutex_lock called on a destroyed mutex (0xb4000076e489ff48)
I  releaseBufferCallbackThunk bufferId:111351322116098 framenumber:166 blastBufferQueue is dead
A  Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 26137 (MHD-single), pid 25926 (wix.kiwixmobile)
I  Explicit concurrent copying GC freed 2515(195KB) AllocSpace objects, 0(0B) LOS objects, 49% free, 8689KB/16MB, paused 101us,412us total 52.866ms
A  Cmdline: org.kiwix.kiwixmobile
A  pid: 25926, tid: 26137, name: MHD-single  >>> org.kiwix.kiwixmobile <<<
A        #04 pc 0000000000091b6c  /data/app/~~X7NK_kCDidpi-kQSDVS7AQ==/org.kiwix.kiwixmobile-XyshDZmEe1OjomDoUdvZnA==/base.apk!libc++_shared.so (std::__ndk1::mutex::lock()+8) (BuildId: ece72a2ebc3774a1be9fd21271258acd3bcdfaa7)
A        #05 pc 0000000000176b84  /data/app/~~X7NK_kCDidpi-kQSDVS7AQ==/org.kiwix.kiwixmobile-XyshDZmEe1OjomDoUdvZnA==/base.apk!libkiwix.so (kiwix::Library::getRevision() const+16)
A        #06 pc 00000000001c3ebc  /data/app/~~X7NK_kCDidpi-kQSDVS7AQ==/org.kiwix.kiwixmobile-XyshDZmEe1OjomDoUdvZnA==/base.apk!libkiwix.so (kiwix::InternalServer::getLibraryId() const+48)
A        #07 pc 00000000001c337c  /data/app/~~X7NK_kCDidpi-kQSDVS7AQ==/org.kiwix.kiwixmobile-XyshDZmEe1OjomDoUdvZnA==/base.apk!libkiwix.so (kiwix::InternalServer::handle_request(kiwix::RequestContext const&)+60)
A        #08 pc 00000000001c303c  /data/app/~~X7NK_kCDidpi-kQSDVS7AQ==/org.kiwix.kiwixmobile-XyshDZmEe1OjomDoUdvZnA==/base.apk!libkiwix.so (kiwix::InternalServer::handlerCallback(MHD_Connection*, char const*, char const*, char const*, char const*, unsigned long*, void**)+796)
A        #09 pc 00000000004e8740  /data/app/~~X7NK_kCDidpi-kQSDVS7AQ==/org.kiwix.kiwixmobile-XyshDZmEe1OjomDoUdvZnA==/base.apk!libkiwix.so
D  Watching instance of androidx.fragment.app.FragmentManagerViewModel (androidx.fragment.app.FragmentManagerViewModel received ViewModel#onCleared() callback) with key 4f7e5630-a88e-4905-aea2-84d776dbb019
D  Watching instance of androidx.lifecycle.SavedStateHandlesVM (androidx.lifecycle.SavedStateHandlesVM received ViewModel#onCleared() callback) with key 7a8fb9c1-43a8-4fcd-a8a5-b37134282617
---------------------------- PROCESS ENDED (25926) for package org.kiwix.kiwixmobile ----------------------------

If we look at the logs it indicate the crash happen in the following method:

(kiwix::Library::getRevision() const+16)
(kiwix::InternalServer::getLibraryId() const+48)
(kiwix::InternalServer::handle_request(kiwix::RequestContext const&)+60)
(kiwix::InternalServer::handlerCallback(MHD_Connection*, char const*, char const*, char const*, char const*, unsigned long*, void**)+796)

[MohitMaliFtechiz]

@mgautierfr Thanks for your fix https://github.com/kiwix/kiwix-android/pull/3467, we have adapted those changes in https://github.com/kiwix/kiwix-android/pull/3217 so I'm closing this ticket.