remote-android / redroid-doc

redroid (Remote-Android) is a multi-arch, GPU enabled, Android in Cloud solution. Track issues / docs here
4.11k stars 295 forks source link

device offline in WSL2 5.10.74.3-microsoft-standard-WSL2 #68

Closed vanokh closed 2 years ago

vanokh commented 2 years ago

Hi, I'm running docker 20.10.11 in Windows 10 using WSL2 Ubuntu. Kernel 5.10.74.3-microsoft-standard-WSL2 rebuilt including ashmem and binder:

user@win10:~/redroid$ grep binder /proc/filesystems
nodev   binder
user@win10:~/redroid$ grep ashmem /proc/misc
 63 ashmem
user@win10:~/redroid$ ls /dev/binderfs
binder  binder-control  hwbinder  vndbinder

Docker is started using: docker run -itd --rm --memory-swappiness=0 --privileged --pull always -v ~/data:/data -p 5555:5555 redroid/redroid:11.0.0-latest redroid.gpu.mode=guest

Adb cannot connect and shows device offline.

ps -A output:

USER            PID   PPID     VSZ    RSS WCHAN            ADDR S NAME
root              1      0 10781300  8924 do_epoll_+          0 S init
root             10      1 10760760  6504 __x64_sys+          0 S init
root             12      1 10761980  7200 __x64_sys+          0 S ueventd
logd             25      1 10762208  5488 sigsuspend          0 S logd
lmkd             26      1 10756580  2500 do_epoll_+          0 S lmkd
system           27      1 10759524  5208 0                   0 R servicemanager
system           28      1 10761284  6348 do_epoll_+          0 S hwservicemanager
shell            29      1 10755144  2692 0                   0 S sh
root             30      1 10768236  8128 binder_th+          0 S vold
system           35      1 10762468  4960 binder_th+          0 S android.system.suspend@1.0-service
nobody           36      1 10762240  6908 binder_th+          0 S android.hardware.keymaster@3.0-service
tombstoned       53      1 10755448  2248 do_epoll_+          0 S tombstoned
system           59      1 10758740  4320 binder_th+          0 S android.hidl.allocator@1.0-service
audioserver      60      1   29012   9204 binder_th+          0 S android.hardware.audio.service
media            61      1   19576   6400 binder_th+          0 S android.hardware.cas@1.2-service
system           62      1 10761260  5844 binder_th+          0 S android.hardware.gatekeeper@1.0-service.software
system           63      1 10761252  5196 binder_th+          0 S android.hardware.graphics.allocator@2.0-service
system           64      1 10773120 11076 binder_th+          0 S android.hardware.graphics.composer@2.1-service
system           65      1 10760428  5424 do_epoll_+          0 S android.hardware.health@2.1-service
system           66      1 10759880  5056 binder_th+          0 S android.hardware.power.stats@1.0-service.mock
wifi             67      1 10764012  8440 binder_th+          0 S android.hardware.wifi@1.0-service
nobody           69      1 10759372  4416 binder_th+          0 S android.hardware.power-service.example
audioserver      71      1 10798440 21628 binder_th+          0 S audioserver
credstore        72      1 10764476  7964 binder_th+          0 S credstore
gpu_service      73      1 10762724  6896 binder_th+          0 S gpuservice
system           75      1 10848176 43324 do_epoll_+          0 S surfaceflinger
shell            86      1 10766932  5936 do_epoll_+          0 S adbd
nobody           91      1 10757556  3216 __x64_sys+          0 S traced_probes
nobody           92      1 10757556  3428 __x64_sys+          0 S traced
root             93      1 13533048 200240 __x64_sys+         0 S zygote64
root            101      1 1879132 189744 __ia32_co+          0 S zygote
cameraserver    104      1   58052  21148 binder_th+          0 S cameraserver
drm             115      1   26228   7916 binder_th+          0 S drmserver
system          116      1 10759816  4916 binder_th+          0 S idmap2d
incidentd       117      1 10762032  5260 do_epoll_+          0 S incidentd
root            118      1 10761704  5208 binder_th+          0 S installd
keystore        119      1 10764944  7964 binder_th+          0 S keystore
mediaex         123      1 10793124 24872 binder_th+          0 S media.extractor
media           124      1 10764640  5720 binder_th+          0 S media.metrics
media           125      1   83520  21504 binder_th+          0 S mediaserver
root            130      1 10782736  9436 binder_th+          0 S netd
root            135      1 10762248  5436 binder_th+          0 S storaged
wifi            142      1 10765868  7616 do_epoll_+          0 S wificond
mediacodec      147      1   53944  17744 binder_th+          0 S media.codec
radio           148      1 10763416  5544 hrtimer_n+          0 S rild
mediacodec      150      1 10792632 19752 binder_th+          0 S media.swcodec
statsd          152      1 10766208  4960 do_epoll_+          0 S statsd
mdnsr           158      1 8652312   1028 0                   0 S mdnsd
system          161      1 10764056  6808 binder_th+          0 S gatekeeperd
root            162    130 10755704  2444 pipe_read           0 S iptables-restore
root            163    130 10755704  2316 pipe_read           0 S ip6tables-restore
system          199     93 13903592 312944 0                  0 S system_server
root            243      0 10755144  2712 sigsuspend          0 S sh
network_sta+    440     93 13091332 122912 do_epoll_+         0 S com.android.networkstack.process
secure_elem+    463     93 13084428 103128 do_epoll_+         0 S com.android.se
system          513     93 13150148 156028 0                  0 S com.android.settings
radio           611     93       0      0 0                   0 R [m.android.phone]
root            629    243 10758500  3388 0                   0 R ps

Logcat logcat.txt

zhouziyang commented 2 years ago

seems something wrong with your data partition, can you mount with another folder (-v ~/data2:/data)? I can reproduce this issue (ethernet network in Android container). I will check this issue. previously, I tested in kernel 5.4; and it worked.

zhouziyang commented 2 years ago

please make sure the following IPV6 features enabled: Networking support > Networking options > The IPv6 protocol IPv6: Router Preference (RFC 4191) support IPv6: Route Information (RFC 4191) support IPv6: Multiple Routing Tables IPv6: source address based routing

vanokh commented 2 years ago

I've enabled all mentioned IPV6 features, but still no success, new logcat: logcat_2112.txt

zhouziyang commented 2 years ago

code branch: refs/tags/linux-msft-wsl-5.10.74.3, I change the following configs, and redroid works.

$ diff .config Microsoft/config-wsl
11,12d10
< CONFIG_CC_CAN_LINK=y
< CONFIG_CC_CAN_LINK_STATIC=y
967,968c965
< CONFIG_IPV6_ROUTER_PREF=y
< CONFIG_IPV6_ROUTE_INFO=y
---
> # CONFIG_IPV6_ROUTER_PREF is not set
980,981c977
< CONFIG_IPV6_MULTIPLE_TABLES=y
< CONFIG_IPV6_SUBTREES=y
---
> # CONFIG_IPV6_MULTIPLE_TABLES is not set
1854d1849
< # CONFIG_NET_VRF is not set
2102c2097
< CONFIG_SPMI=m
---
> # CONFIG_SPMI is not set
2631,2656c2626
< CONFIG_STAGING=y
< # CONFIG_COMEDI is not set
< # CONFIG_RTS5208 is not set
< # CONFIG_STAGING_MEDIA is not set
<
< #
< # Android
< #
< CONFIG_ASHMEM=y
< # CONFIG_ION is not set
< # end of Android
<
< # CONFIG_LTE_GDM724X is not set
< # CONFIG_GS_FPGABOOT is not set
< # CONFIG_UNISYSSPAR is not set
<
< #
< # Gasket devices
< #
< # CONFIG_STAGING_GASKET_FRAMEWORK is not set
< # end of Gasket devices
<
< # CONFIG_FIELDBUS_DEV is not set
< # CONFIG_KPC2000 is not set
< # CONFIG_QLGE is not set
< # CONFIG_SPMI_HISI3670 is not set
---
> # CONFIG_STAGING is not set
2798,2802c2768
< CONFIG_ANDROID=y
< CONFIG_ANDROID_BINDER_IPC=y
< CONFIG_ANDROID_BINDERFS=y
< CONFIG_ANDROID_BINDER_DEVICES="binder,hwbinder,vndbinder"
< # CONFIG_ANDROID_BINDER_IPC_SELFTEST is not set
---
> # CONFIG_ANDROID is not set
2821d2786
< # CONFIG_NVMEM_SPMI_SDAM is not set

BTW, can you double check via zcat /proc/config.gz | grep IPV6? and I can not found any useful information in your logcat.txt, you may need grab at the begining (otherwise, useful information maybe lost)

vanokh commented 2 years ago

Branch is the same 5.10.74.3. I've checked the config, IPV6 was there, but these were missing < CONFIG_CC_CAN_LINK=y < CONFIG_CC_CAN_LINK_STATIC=y < CONFIG_SPMI=m added them and rebuild, but again no luck...

$ zcat /proc/config.gz | grep IPV6
CONFIG_IPV6=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
CONFIG_IPV6_OPTIMISTIC_DAD=y
# CONFIG_IPV6_MIP6 is not set
# CONFIG_IPV6_ILA is not set
# CONFIG_IPV6_VTI is not set
CONFIG_IPV6_SIT=y
# CONFIG_IPV6_SIT_6RD is not set
CONFIG_IPV6_NDISC_NODETYPE=y
# CONFIG_IPV6_TUNNEL is not set
CONFIG_IPV6_MULTIPLE_TABLES=y
CONFIG_IPV6_SUBTREES=y
# CONFIG_IPV6_MROUTE is not set
# CONFIG_IPV6_SEG6_LWTUNNEL is not set
# CONFIG_IPV6_SEG6_HMAC is not set
# CONFIG_IPV6_RPL_LWTUNNEL is not set
# CONFIG_IP_VS_IPV6 is not set
# CONFIG_NF_SOCKET_IPV6 is not set
# CONFIG_NF_TPROXY_IPV6 is not set
CONFIG_NF_TABLES_IPV6=y
# CONFIG_NFT_DUP_IPV6 is not set
# CONFIG_NFT_FIB_IPV6 is not set
# CONFIG_NF_DUP_IPV6 is not set
CONFIG_NF_REJECT_IPV6=y
CONFIG_NF_LOG_IPV6=y
CONFIG_IP6_NF_MATCH_IPV6HEADER=y
CONFIG_NF_DEFRAG_IPV6=y
$diff .config Microsoft/config-wsl
5c5
< CONFIG_CC_VERSION_TEXT="gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0"
---
> CONFIG_CC_VERSION_TEXT="x86_64-msft-linux-gcc (GCC) 9.3.0"
11,12d10
< CONFIG_CC_CAN_LINK=y
< CONFIG_CC_CAN_LINK_STATIC=y
967,968c965
< CONFIG_IPV6_ROUTER_PREF=y
< CONFIG_IPV6_ROUTE_INFO=y
---
> # CONFIG_IPV6_ROUTER_PREF is not set
980,981c977
< CONFIG_IPV6_MULTIPLE_TABLES=y
< CONFIG_IPV6_SUBTREES=y
---
> # CONFIG_IPV6_MULTIPLE_TABLES is not set
1854d1849
< # CONFIG_NET_VRF is not set
2102c2097
< CONFIG_SPMI=m
---
> # CONFIG_SPMI is not set
2631,2656c2626
< CONFIG_STAGING=y
< # CONFIG_COMEDI is not set
< # CONFIG_RTS5208 is not set
< # CONFIG_STAGING_MEDIA is not set
<
< #
< # Android
< #
< CONFIG_ASHMEM=y
< # CONFIG_ION is not set
< # end of Android
<
< # CONFIG_LTE_GDM724X is not set
< # CONFIG_GS_FPGABOOT is not set
< # CONFIG_UNISYSSPAR is not set
<
< #
< # Gasket devices
< #
< # CONFIG_STAGING_GASKET_FRAMEWORK is not set
< # end of Gasket devices
<
< # CONFIG_FIELDBUS_DEV is not set
< # CONFIG_KPC2000 is not set
< # CONFIG_QLGE is not set
< # CONFIG_SPMI_HISI3670 is not set
---
> # CONFIG_STAGING is not set
2798,2802c2768
< CONFIG_ANDROID=y
< CONFIG_ANDROID_BINDER_IPC=y
< CONFIG_ANDROID_BINDERFS=y
< CONFIG_ANDROID_BINDER_DEVICES="binder,hwbinder,vndbinder"
< # CONFIG_ANDROID_BINDER_IPC_SELFTEST is not set
---
> # CONFIG_ANDROID is not set
2821d2786
< # CONFIG_NVMEM_SPMI_SDAM is not set
3523c3488,3494
< # CONFIG_DEBUG_INFO is not set
---
> CONFIG_DEBUG_INFO=y
> # CONFIG_DEBUG_INFO_REDUCED is not set
> # CONFIG_DEBUG_INFO_COMPRESSED is not set
> # CONFIG_DEBUG_INFO_SPLIT is not set
> # CONFIG_DEBUG_INFO_DWARF4 is not set
> CONFIG_DEBUG_INFO_BTF=y
> # CONFIG_GDB_SCRIPTS is not set

I've tried to catch logcat as early as possible - it shows even start of the Linux kernel logcat2212_1.txt a bit later it shows only the same errors with com.android.phone as first time logcat2212_2.txt

zhouziyang commented 2 years ago

there are many migrate errors.. did you run redroid with different version, but mount the same data partition? try mount a new folder -v ~/data2:/data?

12-22 09:11:15.818   195   219 W PackageManager: Failed to migrate com.android.theme.icon_pack.filled.settings: android.os.ServiceSpecificException: Failed to mark default storage /data/data/com.android.theme.icon_pack.filled.settings (code 95)
12-22 09:11:15.825   195   219 W PackageManager: Failed to migrate com.android.dreams.basic: android.os.ServiceSpecificException: Failed to mark default storage /data/data/com.android.dreams.basic (code 95)
12-22 09:11:15.826   195   219 W PackageManager: Failed to migrate com.android.theme.icon_pack.kai.systemui: android.os.ServiceSpecificException: Failed to mark default storage /data/data/com.android.theme.icon_pack.kai.systemui (code 95)
12-22 09:11:15.829   195   219 W PackageManager: Failed to migrate com.android.webview: android.os.ServiceSpecificException: Failed to mark default storage /data/data/com.android.webview (code 95)
12-22 09:11:15.830   195   219 W PackageManager: Failed to migrate com.android.se: android.os.ServiceSpecificException: Failed to mark default storage /data/data/com.android.se (code 95)
12-22 09:11:15.831   195   219 W PackageManager: Failed to migrate com.android.bips: android.os.ServiceSpecificException: Failed to mark default storage /data/data/com.android.bips (code 95)
vanokh commented 2 years ago

I did run first time with old kernel, but I don't mount anything manually. Can you tell me how I can clean up partition or remount a new one?

zhouziyang commented 2 years ago

try run docker run ... -v ~/some-dir:/data ...

docker run -itd --rm --memory-swappiness=0 --privileged \
    --pull always \
    -v ~/some-dir:/data \
    -p 5555:5555 \
    redroid/redroid:11.0.0-latest
vanokh commented 2 years ago

this time tried with

docker run -itd --rm --memory-swappiness=0 --privileged \
    --pull always \
    -v ~/data2:/data \
    -p 5555:5555 \
    redroid/redroid:11.0.0-latest

no migration errors, but the same error in the end logcat2212_3.txt

do I need to prepare the data dir or docker make it itself? could it be some issue with binder? there is e.g. following error in logcat:

12-22 11:07:16.353   140   140 I ServiceManager: Waiting for service 'statscompanion' on '/dev/binder'...
12-22 11:07:16.454   140   140 I ServiceManager: Waiting for service 'statscompanion' on '/dev/binder'...
12-22 11:07:16.555   140   140 W ServiceManager: Service statscompanion didn't start. Returning NULL

I did only in WSL

sudo mkdir /dev/binderfs
sudo mount -t binder binder /dev/binderfs

and checked that there are devices

$ ls /dev/binderfs
binder  binder-control  hwbinder  vndbinder
zhouziyang commented 2 years ago

you don't need mount binderfs in host side (redroid will mount by itself) weired, com.android.phone cannot boot; and this prevent the following network stack booting.

As a workaround, try mkdir -p /data/user_de/0/com.android.providers.telephony/databases && chmod 777 /data/user_de/0/com.android.providers.telephony/databases

12-22 11:07:36.269  3849  3849 E AndroidRuntime: java.lang.RuntimeException: Unable to get provider com.android.providers.telephony.CarrierIdProvider: android.database.sqlite.SQLiteCantOpenDatabaseException: Cannot open database '/data/user_de/0/com.android.providers.telephony/databases/carrierIdentification.db': Directory /data/user_de/0/com.android.providers.telephony/databases doesn't exist
12-22 11:07:36.269  3849  3849 E AndroidRuntime:        at android.app.ActivityThread.installProvider(ActivityThread.java:7251)
12-22 11:07:36.269  3849  3849 E AndroidRuntime:        at android.app.ActivityThread.installContentProviders(ActivityThread.java:6787)
12-22 11:07:36.269  3849  3849 E AndroidRuntime:        at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6704)
12-22 11:07:36.269  3849  3849 E AndroidRuntime:        at android.app.ActivityThread.access$1300(ActivityThread.java:237)
12-22 11:07:36.269  3849  3849 E AndroidRuntime:        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913)
12-22 11:07:36.269  3849  3849 E AndroidRuntime:        at android.os.Handler.dispatchMessage(Handler.java:106)
vanokh commented 2 years ago

I've tried mkdir, but then /data/user_de/0 is deleted shortly after:

...
12-22 11:37:54.438  2597  2597 F zygote64: jni_internal.cc:729] JNI FatalError called: (android.ext.services) frameworks/base/core/jni/com_android_internal_os_Zygote.cpp:795: Failed to mount tmpfs to /data/data: No such file or directory
...
12-22 11:37:54.533   196   220 W PackageManager: Destroying user 0 on volume null because we failed to prepare: java.io.IOException: getxattr failed: ENOTSUP (Operation not supported on transport endpoint)

logcat2212_4.txt

zhouziyang commented 2 years ago

folder ~/data2 is located in nfs?

try -v <some-folder-in-ext4-instead-of-nfs>:/data

/data should supoort xattr, this required by Android system.

vanokh commented 2 years ago

do I need to create it in WSL? because there is no any ~/data in WSL

zhouziyang commented 2 years ago

you don't need create folder by yourself (docker will help to)

vanokh commented 2 years ago

the root of WSL is ext4 for sure and the ~ should be as well. sorry, I don't know docker so well and how it works in WSL

vanokh commented 2 years ago

I've created new volume using docker volume create my-data and then started docker with --mount source=my-data,target=/data. now I can connect with adb, I will test more later and let you know

vanokh commented 2 years ago

yes, it works now! :) successfully connected with scrcpy I didn't understand, why docker cannot use ~/data, but creating the volume resolved the problem. thanks for the support! I guess, the configuration of newer kernels of WSL should be included in documentation

zhouziyang commented 2 years ago

thanks for your feedback, I will update doc soon.

eritpchy commented 2 years ago

please make sure the following IPV6 features enabled: Networking support > Networking options > The IPv6 protocol IPv6: Router Preference (RFC 4191) support IPv6: Route Information (RFC 4191) support IPv6: Multiple Routing Tables IPv6: source address based routing

thanks, https://github.com/eritpchy/WSL2-Linux-Kernel-Docker-Build/commit/48131f8e75a2548ab3e523a6260863e444985e6f

CMingTseng commented 2 years ago

yes, it works now! :) successfully connected with scrcpy I didn't understand, why docker cannot use ~/data, but creating the volume resolved the problem. thanks for the support! I guess, the configuration of newer kernels of WSL should be included in documentation

Dear Sir

do you have step by step docs?

i also want run redroid @WSL2

and

do you add Gapps & magisk ?

THX

zhouziyang commented 2 years ago

@CMingTseng try the links provided by @eritpchy , need compile linux kernel. you can add GApps via rebuilding from source or add GApps blobs into redroid directly. Magisk not supported (feasible, need integrate source into redroid and rebuild)