Closed yevh-berdnyk closed 10 months ago
@yakimant @siddarthkay
May be you have an understanding how we can get a compatible build to try it out?
Thank you in advance for your help!
Hi @churik : could you see if installing PRs from https://github.com/status-im/status-mobile/pull/17241 works? We've bumped up the SDKVersions in the upgrade. ref : https://ci.status.im/job/status-mobile/job/prs/job/android-e2e/job/PR-17241/12/artifact/result/StatusIm-Mobile-230914-143157-1b2bed-pr17241-x86.apk
Hi @churik : could you see if installing PRs from #17241 works? We've bumped up the SDKVersions in the upgrade. ref : https://ci.status.im/job/status-mobile/job/prs/job/android-e2e/job/PR-17241/12/artifact/result/StatusIm-Mobile-230914-143157-1b2bed-pr17241-x86.apk
Hi @siddarthkay, I've just tried installing this apk on local Android12 (x86_64) emulator and having the same issue - no matching abis. The same result with the latest apk from the mentioned PR
Maybe we need to switch to x86_64
for e2e.
As I see images starting with android-31 (Android 12) don't support x86
and maybe they expect binaries to be x86_64
too:
❯ sdkmanager --list | grep system-images | grep "google_apis;"| grep x86 | awk '{print $1}' | sort | uniq
system-images;android-10;google_apis;x86
system-images;android-15;google_apis;x86
system-images;android-16;google_apis;x86
system-images;android-17;google_apis;x86
system-images;android-18;google_apis;x86
system-images;android-19;google_apis;x86
system-images;android-21;google_apis;x86
system-images;android-21;google_apis;x86_64
system-images;android-22;google_apis;x86
system-images;android-22;google_apis;x86_64
system-images;android-23;google_apis;x86
system-images;android-23;google_apis;x86_64
system-images;android-24;google_apis;x86
system-images;android-24;google_apis;x86_64
system-images;android-25;google_apis;x86
system-images;android-25;google_apis;x86_64
system-images;android-26;google_apis;x86
system-images;android-26;google_apis;x86_64
system-images;android-27;google_apis;x86
system-images;android-28;google_apis;x86
system-images;android-28;google_apis;x86_64
system-images;android-29;google_apis;x86
system-images;android-29;google_apis;x86_64
system-images;android-30;google_apis;x86
system-images;android-30;google_apis;x86_64
system-images;android-31;google_apis;x86_64
system-images;android-32;google_apis;x86_64
system-images;android-33;google_apis;x86_64
system-images;android-34;google_apis;x86_64
Let me create a buid for x86_64
to test.
@churik, can you please try this one: https://ci.status.im/job/status-mobile/job/prs/job/android-e2e/job/PR-17335/1/artifact/result/StatusIm-Mobile-230919-125807-11897b-pr17335-x86_64.apk
I am on ARM cpu, so x86_64 doesn't run without additional setup unfortunately.
hi @yakimant : same thing for me back when I had tried : x86_64 doesn't run without additional setup unfortunately.
I will try on my linux machine tomorrow
This can be tested locally too:
x86_64
emulator (check with avdmanager list avd
)make run-android
should build for x86_64
(see the first lines of log)
If not - you can also run ANDROID_ABI_INCLUDE=x86_64 make run-android
to enforce build arch.@siddarthkay, can you please try to reproduce it locally?
So, with builds from https://github.com/status-im/status-mobile/issues/17265#issuecomment-1725478295 I'm able to install apk, but it fails after creating the profile (reproduced on both Android 12 and Android 10)
Logcat: logcat (26).log
It is blocker for e2e. Can you please assist @siddarthkay ?
Sure I will @churik ! I'm afk today but will be able to take a look first thing tomorrow 👍🏻
First look at logcat and this seems to be relevant part of crash log (Keeping it very verbose for now)
--------- beginning of crash
09-19 13:22:02.386 6214 6327 F libc : Fatal signal 31 (SIGSYS), code 1 (SYS_SECCOMP) in tid 6327 (create_react_co), pid 6214 (tus.ethereum.pr)
09-19 13:22:02.434 763 939 D EGL_emulation: app_time_stats: avg=15518.90ms min=15518.90ms max=15518.90ms count=1
09-19 13:22:02.450 6739 6739 I crash_dump64: obtaining output fd from tombstoned, type: kDebuggerdTombstoneProto
09-19 13:22:02.451 294 294 I tombstoned: received crash request for pid 6327
09-19 13:22:02.452 6739 6739 I crash_dump64: performing dump of process 6214 (target tid = 6327)
09-19 13:22:02.459 6739 6739 E DEBUG : failed to read /proc/uptime: Permission denied
09-19 13:22:02.517 6739 6739 W unwind : Failed to initialize DEX file support: dlopen failed: library "libdexfile.so" not found
09-19 13:22:02.552 6142 6188 I appium : channel read: POST /session/eaf577e7-fbe0-445c-84cc-2c2ae7edc0a5/element
09-19 13:22:02.552 6142 6188 I appium : FindElement command
09-19 13:22:02.553 6142 6188 I appium : method: 'xpath', selector: '//*[@text="Maybe later"]'
09-19 13:22:02.554 6142 6188 I appium : Waiting up to 10000ms for the device to idle
09-19 13:22:02.754 6739 6739 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
09-19 13:22:02.754 6739 6739 F DEBUG : Build fingerprint: 'google/sdk_gphone64_x86_64/emulator64_x86_64_arm64:12/SE1A.220826.008/10564458:userdebug/dev-keys'
09-19 13:22:02.754 6739 6739 F DEBUG : Revision: '0'
09-19 13:22:02.754 6739 6739 F DEBUG : ABI: 'x86_64'
09-19 13:22:02.754 6739 6739 F DEBUG : Timestamp: 2023-09-19 13:22:02.459559125+0000
09-19 13:22:02.754 6739 6739 F DEBUG : Process uptime: 0s
09-19 13:22:02.754 6739 6739 F DEBUG : Cmdline: im.status.ethereum.pr
09-19 13:22:02.754 6739 6739 F DEBUG : pid: 6214, tid: 6327, name: create_react_co >>> im.status.ethereum.pr <<<
09-19 13:22:02.754 6739 6739 F DEBUG : uid: 10149
09-19 13:22:02.754 6739 6739 F DEBUG : signal 31 (SIGSYS), code 1 (SYS_SECCOMP), fault addr --------
09-19 13:22:02.754 6739 6739 F DEBUG : Cause: seccomp prevented call to disallowed x86_64 system call 232
09-19 13:22:02.754 6739 6739 F DEBUG : rax 00000000000000e8 rbx 00000000000000c0 rcx 000077f062a8f48e rdx 0000000000000001
09-19 13:22:02.754 6739 6739 F DEBUG : r8 0000000000000000 r9 0000000000000000 r10 ffffffffffffffff r11 0000000000000202
09-19 13:22:02.754 6739 6739 F DEBUG : r12 0000000000000000 r13 0000000000000001 r14 000000c000603520 r15 0000000000000001
09-19 13:22:02.754 6739 6739 F DEBUG : rdi 00000000000000c0 rsi 000000c000a60f84
09-19 13:22:02.754 6739 6739 F DEBUG : rbp 000000c000a60e08 rsp 000000c000a60dc8 rip 000077f062a8f48e
09-19 13:22:02.754 6739 6739 F DEBUG : backtrace:
09-19 13:22:02.754 6739 6739 F DEBUG : #00 pc 00000000012ef48e /data/app/~~T7K40a971VhyO8hwRX-9Ow==/im.status.ethereum.pr-WWr32Mrjmjy4CYqwDI9a3Q==/lib/x86_64/libgojni.so
09-19 13:22:02.780 294 294 E tombstoned: Tombstone written to: tombstone_01
09-19 13:22:02.780 588 6744 I am_crash: [588,0,im.status.ethereum.pr,951598660,Native crash,Bad system call,unknown,0]
09-19 13:22:02.781 588 6745 I DropBoxManagerService: add tag=data_app_native_crash isTagEnabled=true flags=0x2
09-19 13:22:02.782 588 6744 W ActivityTaskManager: Force finishing activity im.status.ethereum.pr/im.status.ethereum.MainActivity
09-19 13:22:02.782 588 6744 I wm_finish_activity: [0,153421236,20,im.status.ethereum.pr/im.status.ethereum.MainActivity,force-crash]
maybe something to do with method: 'xpath', selector: '//*[@text="Maybe later"]'
investigating further
I pinned the builds to keep the artifacts from cleaning in case you will need them:
2. run
ANDROID_ABI_INCLUDE=x86_64 make run-android
This works for me on linux and I was able to run the app. The builds do get generated on mac, however I can not get an x86_64 emulator to work on arm mac.
I was able to reproduce the crash and it seems to be some very weird error.
2023-09-24 22:13:57.147 10125-10295 libc
im.status.ethereum.debug
A Fatal signal 31 (SIGSYS), code 1 (SYS_SECCOMP), syscall 232 in tid 10295 (create_react_co), pid 10125 (.ethereum.debug)
Flipper also saw this crash on x86_64 simulators , ref : https://github.com/facebook/flipper/issues/482 It seems they fixed it with bumping ndkVersions. Maybe that can also help us. Investigating further.
maybe something to do with
method: 'xpath', selector: '//*[@text="Maybe later"]'
Update on this statement : The issue has nothing to do with e2e code, the app crashes on x86_64 emulators
The error we are interested is documented here : https://source.android.com/docs/core/tests/debug/native-crash#seccomp
We can probably ask Android to chill with the illegal Syscall for a bit in debug environments like this :
adb shell setenforce 0 && adb stop && adb start
Another error which seems highly suspicious is this one 09-25 07:03:25.267 14380 14544 E GoLog : [0925/070325.267701:ERROR:elf_dynamic_array_reader.h(64)] tag not found
which seems to be a known error when apps crash in a linux environment, ref vscode : https://github.com/microsoft/vscode/issues/180648
adb shell setenforce 0 && adb stop && adb start
This did not help much and even if it did it won't be of any use since e2e builds would not be debug variants. I have a hunch that says this crash happens because of either of the 2 libraries :
react-native-reanimated
okhttp
But I don't have proof yet, The way forward should be to comment out cljs code right after set password screen and try to narrow down the function call which triggers this crash.
The crash starts with:
F libc : Fatal signal 31 (SIGSYS), code 1 (SYS_SECCOMP) in tid 6327 (create_react_co), pid 6214 (tus.ethereum.pr)
According to this post:
Android 8 O (SDK 26) limits which system calls are allowed for security reasons by enabling a feature called secure computing in the Linux kernel.
This means only whitelisted calls can be executed and that any other call will result in signal 31 (SIGSYS), code 1 (SYS_SECCOMP), like you are experiencing. You will need to examine the stack trace of this signal to find out which system call was not allowed (which was not listed completely in your question).
You can find a list of allowed calls here. Any other call is not allowed.
And based on this line:
Cause: seccomp prevented call to disallowed x86_64 system call 232
It seems pretty obvious that the issue indeed is the Seccomp filter: https://android-developers.googleblog.com/2017/07/seccomp-filter-in-android-o.html
I thought we already use ABI version 31: https://github.com/status-im/status-mobile/blob/2df7a7cf6d46c8d1add73b8965ce8b04e6f7d014/android/gradle.properties#L26-L27
And I saw E2E test run and mostly succeed, so I'm confused. When does this issue manifest?
Noting down the flow of what happens when we press the confirm password button.
:onboarding-2/password-set
re-frame event is dispatched.:onboarding-2/password-set
lives here src/status_im2/contexts/onboarding/events.cljs
and calculates if biometric authentication is possible or not. If it is possible then it dispatches [:navigate-to-within-stack [:enable-biometrics :new-to-status]]
or it dispatches [:onboarding-2/create-account-and-login]
[:onboarding-2/create-account-and-login]
gets dispatched. this event is also defined in the same file (src/status_im2/contexts/onboarding/events.cljs
).[:navigate-to-within-stack [:generating-keys :new-to-status]]
is dispatched and 7 seconds later [:onboarding-2/navigate-to-identifiers]
is dispatched. 7 seconds because it takes that long for the animation to play :)(profile.recover/recover-profile-and-login profile)
otherwise the next call is to (profile.create/create-profile-and-login profile)
(profile.create/create-profile-and-login profile)
is present at src/status_im2/contexts/profile/create/events.cljs
and what happens here is that we call a native module function ->createAccountAndLogin
which is defined at
modules/react-native-status/android/src/main/java/im/status/ethereum/module/StatusModule.java
The response from native module call is "Geth node started" which proves that there was no exception thrown while executing this code.
Shortly after this log message the app crashes.It does not wait for the entire 7 seconds, the crash generally happens within the 1st 2 seconds and almost immediately as we see the Generating Keys Screen.
The crash could be due to the following :
createAccountAndLogin
At this moment I doubt everything and will try to negate assumptions one by one until I can get the crash to stop.
The app does not insta crash which means the x86_64 emulator does not like what ever happens on the Generate Keys screen.
removing function call to createAccountAndLogin
caused the crash to go away. The crash is indeed caused by a panic on status-go side.
Trying to narrow down on the why so that we can tackle this further.
ref : https://github.com/status-im/status-go/compare/develop...login-crash-x86-64
I tried to add some extensive file permissions in a branch I made from the react-native-0.72 upgrade branch here to see if that fixes the issue : https://github.com/status-im/status-mobile/compare/develop...login-crash-x86-64
Turns out the crash was not related to file permissions at OS level but the operations that happened on status-go
side after the keystore
was generated.
I narrowed down the crash to this function call g.store(childAccount, password)
which was here : account/generator/generator.go
I also added extensive logs inside that function ref : https://github.com/status-im/status-go/commit/343e216384d3e372b148df4891ac24cce36298bc to see what went wrong and to my surprise it all looked okay.
I then suspected the root cause to be somewhere in the go-ethereum
code, but I was not too sure.
I had a conversation with @bitgamma and he gave me some awesome insights!
The crash stack trace says : syscall 232
syscall 232
according to docs is epoll_wait
:
https://chromium.googlesource.com/chromiumos/docs/+/master/constants/syscalls.md#x86_64-64_bit:~:text=232-,epoll_wait,-man/%20cs
So most likely the go code which calls epoll_wait
misbehaves and crashes on x86_64 on android 11 onwards.
References to that are : https://github.com/status-im/status-go/blob/develop/rtt/rtt.go#L76 and https://github.com/status-im/tcp-shaker/blob/master/socket_linux.go#L85 cc @jakubgs Now this seems like a good spot to further continue investigation.
possible spots in status-go that make epoll_wait
calls are :
There could be more
This issue at android ndk repo caught my attention : https://github.com/android/ndk/issues/1298
its a seccomp issue with x86_64 emulators which was fixed in ndk r26 and probably backported to ndk r25(not sure) However the error message is not exactly the same as what we face here but I figured its worth a shot to try upgrading ndk to this version to see if it fixes our crash.
we didn't have ndkversion 26 in our nixpkgs so I added them to our fork here : https://github.com/status-im/nixpkgs/commit/cf1c475a5141631ff85d48fbb5e015945df50d37
Unfortunately I couldn't get status-go to build on my mac and it would fail with this error :
> Building status-go for: android/arm,android/arm64,android/386
> /nix/store/m1iynw68yqby0xi18djqfqvqh6z52528-gomobile-unstable-2022-05-18/bin/gomobile:
go build -tags gowaku_skip_migrations,gowaku_no_rln -ldflags -X github.com/status-im/status-
go/params.GitCommit=27b770c41bd2896ebc30b01985eeea1fbe642fba -X github.com/status-im/status-
go/params.IpfsGatewayURL=https://ipfs.status.im/ -X github.com/status-im/status-go/params.Version=0.171.6
-s -w -buildmode=c-shared -o=/private/tmp/nix-build-status-go-0.171.6-27b770c-android.drv-0/gomobile-
work/android/src/main/jniLibs/arm64-v8a/libgojni.so ./gobind failed: exit status 2
> # github.com/status-im/status-go/vendor/github.com/mutecomm/go-sqlcipher/v4
> aesce.c:194:18: error: always_inline function 'vaesimcq_u8' requires target feature 'aes',
but would be inlined into
function 'mbedtls_aesce_inverse_key' that is compiled without support for 'aes'
make: *** [build-android] Error 1
probably @caybro or @alaibe might have an idea on how to get this to work.
I traced down the error to https://github.com/status-im/go-sqlcipher/blob/master/aesce.c#L194 in our fork of go-sqlcipher
package but I wasn't sure If its a build flag issue or something else.
I also tried ndkVersion
25.2.9519653
I do get a successful build, but on x86_64 simulators the app crashes right after generating keys stage which means that the issue is still present :)
The ndk upgrade path was a dead end.
possible spots in status-go that make
epoll_wait
calls are :
patching these libraries with custom build flags to remove syscall for epoll_wait
fixes the crash and is a decent enough solution to move forward.
Bug Report
Problem
To keep our e2e tests up to date we need to use relevant Android version. SauceLabs emulators are x86_64 only. And now it's not possible to install and use e2e application on such emulators with Android 12 or more.
Expected behavior
The e2e apk is installed on Android 12
Actual behavior
Reproduction
Additional Information