Compile/run device mapper user space utils on Android

bamos commented 10 years ago

Further pursue based on the findings in #7.

bamos commented 10 years ago

This repo provides an lvm binary compiled for ARM, and I copied it to my device and it works fine. I put sbin and etc in /data/gray/lvm and set LVM_SYSTEM_DIR=/data/gray/lvm. However, this solution's looking a little buggy.

Since Android doesn't provide a way to create loop devices that I could use to play with lvm, I mounted 2 nbd devices.

1|root@maguro:/data/nbdroid # ./nbd-client 128.2.209.242 9001 /dev/nbd1
Negotiation: ..size = 10MB
bs=1024, sz=10485760 bytes
root@maguro:/data/nbdroid # ./nbd-client 128.2.209.242 9002 /dev/nbd2
Negotiation: ..size = 10MB
bs=1024, sz=10485760 bytes

Next, I had trouble creating an lvm physical volume on nbd1. :(

3|root@maguro:/data/gray/lvm/sbin # ./lvm pvcreate /dev/nbd1
File descriptor 3 (socket:[4194]) leaked on lvm invocation. Parent PID 22330: sh
File descriptor 4 (socket:[201390]) leaked on lvm invocation. Parent PID 22330: sh
File descriptor 5 (/data/data/eu.chainfire.supersu/logs/201410031425290000.GRANTED.ADB shell) leaked on lvm invocation. Parent PID 22330: sh
File descriptor 6 (/dev/pts/1) leaked on lvm invocation. Parent PID 22330: sh
File descriptor 7 (/dev/pts/2) leaked on lvm invocation. Parent PID 22330: sh
File descriptor 9 (/dev/__properties__) leaked on lvm invocation. Parent PID 22330: sh
  /dev/nbd1: read failed after 0 of 1024 at 0: Input/output error
  /dev/nbd1: read failed after 0 of 1024 at 10420224: Input/output error
  /dev/nbd1: read failed after 0 of 1024 at 10477568: Input/output error
  /dev/nbd1: read failed after 0 of 1024 at 4096: Input/output error
  /dev/nbd1: read failed after 0 of 2048 at 0: Input/output error
  /dev/nbd1: read failed after 0 of 1024 at 3072: Input/output error
  Fatal error while trying to detect swap signature on /dev/nbd1.

Output of lvmdiskscan, looks like trying to add nbd1 crashed the nbd connection.

5|root@maguro:/data/gray/lvm/sbin # ./lvm lvmdiskscan
File descriptor 3 (socket:[4194]) leaked on lvm invocation. Parent PID 22330: sh
File descriptor 4 (socket:[201390]) leaked on lvm invocation. Parent PID 22330: sh
File descriptor 5 (/data/data/eu.chainfire.supersu/logs/201410031425290000.GRANTED.ADB shell) leaked on lvm invocation. Parent PID 22330: sh
File descriptor 6 (/dev/pts/1) leaked on lvm invocation. Parent PID 22330: sh
File descriptor 7 (/dev/pts/2) leaked on lvm invocation. Parent PID 22330: sh
File descriptor 9 (/dev/__properties__) leaked on lvm invocation. Parent PID 22330: sh
  /dev/nbd1: read failed after 0 of 1024 at 0: Input/output error
  /dev/nbd1: read failed after 0 of 1024 at 10420224: Input/output error
  /dev/nbd1: read failed after 0 of 1024 at 10477568: Input/output error
  /dev/nbd1: read failed after 0 of 1024 at 4096: Input/output error
  /dev/block/ram0       [       8.00 MiB]
  /dev/block/mmcblk0p8  [      11.94 MiB]
  /dev/nbd1: read failed after 0 of 2048 at 0: Input/output error
  /dev/block/ram1       [       8.00 MiB]
  /dev/block/mmcblk0p9  [      16.00 MiB]
  /dev/block/ram2       [       8.00 MiB]
  /dev/block/mmcblk0p10 [     654.00 MiB]
  /dev/block/mmcblk0p2  [       3.50 MiB]
  /dev/block/ram3       [       8.00 MiB]
  /dev/block/mmcblk0p11 [     432.00 MiB]
  /dev/block/mmcblk0p3  [      20.00 MiB]
  /dev/block/ram4       [       8.00 MiB]
  /dev/block/mmcblk0p12 [      13.54 GiB]
  /dev/block/mmcblk0p4  [       8.00 MiB]
  /dev/block/ram5       [       8.00 MiB]
  /dev/block/mmcblk0p5  [       4.00 MiB]
  /dev/block/ram6       [       8.00 MiB]
  /dev/block/mmcblk0p6  [       4.00 MiB]
  /dev/block/ram7       [       8.00 MiB]
  /dev/block/mmcblk0p7  [       8.00 MiB]
  /dev/block/ram8       [       8.00 MiB]
  /dev/block/ram9       [       8.00 MiB]
  /dev/block/ram10      [       8.00 MiB]
  /dev/block/ram11      [       8.00 MiB]
  /dev/block/ram12      [       8.00 MiB]
  /dev/block/ram13      [       8.00 MiB]
  /dev/block/ram14      [       8.00 MiB]
  /dev/block/ram15      [       8.00 MiB]
  0 disks
  27 partitions
  0 LVM physical volume whole disks
  0 LVM physical volumes

bamos commented 10 years ago

Using the same technique from that repo but building device mapper might work well - will try that later.

wenluhu commented 10 years ago

This guy seems to be able to run "losetup". Not sure what he did. I'm putting it here just to leave a note. Not our highest priority now. https://code.google.com/p/cryptonite/issues/detail?id=53

wenluhu commented 10 years ago

One thought, everything else is in /dev/block/ instead of the usual /dev/. Should we put nbd in /dev/block too?

bamos commented 10 years ago

@wenluhu I wonder if using CWM will let us remount with devmapper, can you try?

http://forum.xda-developers.com/showthread.php?t=1396366

You should be able to boot into CWM, unmount the partitions, connect with adb and run it from there. At least if your version of CWM has fsck in it.

wenluhu commented 10 years ago

Sure. I'll look into this.

wenluhu commented 10 years ago

I have booted into CWM. On the phone, I'm able to mount and unmount /system and /cache, mount /data. Unmounting /data does not work.

Mounting and unmounting through adb works perfectly after

./adb kill-server
sudo ./adb start-server

wenluhu commented 10 years ago

Back to testing LVM, it was hard to test LVM alone, separately from NBD, as we couldn't use loop devices. This post seems to have used loop devices on Android.

busybox losetup /dev/block/loop250 $kit/$userdata”.img”

[x] Checkout busybox

wenluhu commented 10 years ago

busybox losetup works perfectly! It's easy to install busybox through Play Store App Busybox.

shell@maguro:/ $ su
root@maguro:/ # cd data
root@maguro:/data # busybox losetup -a
root@maguro:/data # busybox losetup /dev/block/loop0 loop0.ima
root@maguro:/data # busybox losetup -a                                         
root@maguro:/data # mount -t ext4 /dev/block/loop0 /data/loopmountpoint/     
root@maguro:/data # ls loopmountpoint/                                         
dir_created_in_Ubuntu
lost+found
root@maguro:/data # mount | grep loop
/dev/block/loop0 /data/loopmountpoint ext4 rw,seclabel,relatime,user_xattr,barrier=1,data=ordered 0 0

root@maguro:/data # cd loopmountpoint/                                         
root@maguro:/data/loopmountpoint # mkdir dir_created_on_Android
root@maguro:/data/loopmountpoint # ls
dir_created_on_Android
dir_created_in_Ubuntu
lost+found

root@maguro:/data # busybox umount /dev/block/loop0                          
root@maguro:/data # mount | grep loop

As mkfs.ext4 doesn't work, I need to get the image file from outside.

[x] Copy empty EXT4 image from Ubuntu? It works pretty well so far except kernel panic happened once.
[ ] Download an Android image from the Internet?

TODO

[ ] Test LVM with loop device on Android

wenluhu commented 10 years ago

A quick test on lvm lvcreate --mirrors returned error mirror: Required device-mapper target(s) not detected in your kernel. It seems having lvm installed doesn't necessarily give us device-mapper libraries. We still need to have device-mapper libraries separately for lvm to work.

We have two options according to this post:

[x] Add device-mapper modules. @bamos Have you ever successfully used modprobe on Android? If not, I'm afraid we still need to build device-mapper user space libraries ourselves...
[ ] Upgrade to the latest lvm2

bamos commented 10 years ago

Hi @wenluhu - if I remember correctly, modules are disabled by default in the Linux kernel used for Android. One solution could be to recompile the kernel with modules and device mapper enabled. Does CM have insmod? An alternate could be to just compile the device mapper driver and use insmod. I think this needs a custom kernel in vanilla Android.

wenluhu commented 10 years ago

Ah! I finally learned how to build Android kernel! I haven't experienced such a pain in compiling and building in the past year, as I was mostly using Python and Java. I should feel great when this whole thing starts to work later. :)

I'm trying to build the device mapper kernel module according to this post. The make module step keeps giving me nothing, no .ko files. But by doing just make I got a bunch of .o files. I suspect this is related to that modules are disabled in the kernel.

ubuntu@test:~/omap/drivers/md$ make ARCH=arm CROSS_COMPILE=arm-eabi- -C ~/omap/ M=$PWD modul
es                                                                                          
make: Entering directory `/home/ubuntu/omap'
  Building modules, stage 2.
  MODPOST 0 modules
make: Leaving directory `/home/ubuntu/omap'
ubuntu@test:~/omap/drivers/md$ make ARCH=arm CROSS_COMPILE=arm-eabi- -C ~/omap/ M=$PWD
make: Entering directory `/home/ubuntu/omap'
  CC      /home/ubuntu/omap/drivers/md/dm-uevent.o
  CC      /home/ubuntu/omap/drivers/md/dm.o
/home/ubuntu/omap/drivers/md/dm.c: In function 'split_bvec':
/home/ubuntu/omap/drivers/md/dm.c:1070:3: warning: statement with no effect [-Wunused-value]
/home/ubuntu/omap/drivers/md/dm.c: In function 'clone_bio':
/home/ubuntu/omap/drivers/md/dm.c:1097:3: warning: statement with no effect [-Wunused-value]
  CC      /home/ubuntu/omap/drivers/md/dm-table.o
/home/ubuntu/omap/drivers/md/dm-table.c: In function 'dm_table_set_integrity':
/home/ubuntu/omap/drivers/md/dm-table.c:1188:3: warning: statement with no effect [-Wunused-value]
  CC      /home/ubuntu/omap/drivers/md/dm-target.o
  CC      /home/ubuntu/omap/drivers/md/dm-linear.o
  CC      /home/ubuntu/omap/drivers/md/dm-stripe.o
  CC      /home/ubuntu/omap/drivers/md/dm-ioctl.o
  CC      /home/ubuntu/omap/drivers/md/dm-io.o
  CC      /home/ubuntu/omap/drivers/md/dm-kcopyd.o
  CC      /home/ubuntu/omap/drivers/md/dm-sysfs.o
  LD      /home/ubuntu/omap/drivers/md/dm-mod.o
  CC      /home/ubuntu/omap/drivers/md/dm-crypt.o
  LD      /home/ubuntu/omap/drivers/md/built-in.o
  Building modules, stage 2.
  MODPOST 0 modules
make: Leaving directory `/home/ubuntu/omap'

I tried to insmod the *.o files I have. All of them give me the same error.

insmod: init_module 'dm-io.o' failed (Function not implemented)

A quick search on this error shows that we probably still need modules to be enabled, even if we use insmod instead of modprobe, like in this post and this one as well. I think these posts (including the tutorial I followed) are all written before modules are disabled in kernel. I noticed in the log that modules was not disabled until June 2013.

commit fbc1a9de516ca3539bc28531f452912da284748a
Author: Todd Poynor <toddpoynor@google.com>
Date:   Mon Jun 3 12:21:06 2013 -0700

    arm: OMAP4: tuna: turn off module support

    Turn off kernel module support to meet Android security guidelines.

    Change-Id: If9309ad65dc6dfb1cb6848849dd55390cb438e9a
    Signed-off-by: Todd Poynor <toddpoynor@google.com>

I don't think we should recompile kernels with modules enabled, even if that solves all our problems. They disabled modules for security reasons. If we turn it back on, it would be hard to argue that gammaray-android, with potential security risks, can be used to monitor corporate devices.

I'm kind of lost on what I should try next. @bamos @theonewolf Anything on top of your head that I should try next? Have I confused any concepts that are clear to you guys? It would be great if you can correct me now. :) Thanks a lot!

bamos commented 10 years ago

Hi @wenluhu - not using a kernel with modules enabled is ideal, but I needed to recompile the kernel with modules enabled for nbd and I'm not sure of an easy alternate. I've only ever seen insmod on kernel module objects with a .ko extension btw.

What if you try setting the following in .config and building/flashing the entire device image?

CONFIG_MODULES=y
CONFIG_DM_SNAPSHOT=y
CONFIG_DM_MIRROR=y
CONFIG_DM_RAID=y
CONFIG_DM_ZERO=y
CONFIG_DM_MULTIPATH=y
CONFIG_DM_DELAY=y

Hopefully we'll see dm-mirror* files. Perhaps the following error message you got indicates that enabling the DM_MIRROR option is all we need.

A quick test on lvm lvcreate --mirrors returned error mirror: Required device-mapper target(s) not detected in your kernel

If this doesn't work, do you want to debug in person tomorrow or over the weekend?

wenluhu commented 10 years ago

Okay, if we'll enable modules for nbd anyways, I guess we should do this for device mapper too. I have re-compiled the kernel and it looks good. I'm learning how to flash the entire image now.

ubuntu@test:~/omap$ find . -name *mirror*
./drivers/md/.dm-mirror.o.cmd
./drivers/md/dm-mirror.o
./include/config/dm/mirror.h

BTW, when I run insmod on the phone, it asks me for *.o.

root@maguro:/ # insmod
usage: insmod <module.o>

bamos commented 10 years ago

Strange about insmod on Android, mine says that too, but I use it to load nbd.ko:

root@maguro:/data/nbdroid # insmod
usage: insmod <module.o>
255|root@maguro:/data/nbdroid # insmod nbd.ko
root@maguro:/data/nbdroid # echo $?
0

wenluhu commented 10 years ago

The code @bamos sent me through email works perfectly. I have successfully flashed the new kernel to the phone.

adb reboot bootloader
fastboot flash zimage arch/arm/boot/zImage
fastboot reboot

Though I'm not 100% sure that device-mapper is working on Android now, a quick test of lvm create shows that it's complaining something else instead of device-mapper now. I consider this a good sign. I can start playing with lvm now.

root@maguro:/ # lvm lvcreate -L 500M -m1 --mirrorlog core vg0  /dev/loop1 /dev/loop2
File descriptor 3 (socket:[2487]) leaked on lvm invocation. Parent PID 2854: sh
File descriptor 4 (socket:[11169]) leaked on lvm invocation. Parent PID 2854: sh
File descriptor 5 (/data/data/eu.chainfire.supersu/logs/201411071908320000.GRANTED.ADB shell) leaked on lvm invocation. Parent PID 2854: sh
File descriptor 6 (/dev/pts/1) leaked on lvm invocation. Parent PID 2854: sh
File descriptor 7 (/dev/pts/2) leaked on lvm invocation. Parent PID 2854: sh
File descriptor 9 (/dev/__properties__) leaked on lvm invocation. Parent PID 2854: sh
  Volume group "vg0" not found

wenluhu commented 10 years ago

I failed to use the busybox commands that I used before to prepare loop devices. I have finally found a fix. -o option is needed in losetup, or I'll keep getting mount: Invalid argument even if I have specified the file system type. The offset is manually extracted from parted loop0.ima unit B print as in this post

busybox losetup -o 1048576 /dev/block/loop0 loop0.ima

wenluhu commented 10 years ago

Although I still can't mirror loop devices, I manage to create a mirrored volume on top of /dev/block/ram0 and /dev/block/ram1. This at least proves that lvm is properly installed on Android and ready to use now. Yayyyyy!! And I expect nbd devices to act like ram rather than loop in this context, as lvm lvmdiskscan shows ram and nbd devices, but not loop devices.

A tutorial for basic lvm stuff A tutorial for lvm mirror

root@maguro:/data # lvm pvcreate /dev/block/ram0                             
  Physical volume "/dev/block/ram0" successfully created
root@maguro:/data # lvm pvcreate /dev/block/loop0                              
  Device /dev/block/loop0 not found (or ignored by filtering).
root@maguro:/data # lvm pvcreate /dev/block/ram1                                  
  Physical volume "/dev/block/ram1" successfully created
root@maguro:/data # lvm lvmdiskscan                                            
  /dev/block/ram0       [       8.00 MiB] LVM physical volume
  /dev/block/mmcblk0p8  [      11.94 MiB] 
  /dev/block/ram1       [       8.00 MiB] LVM physical volume
  /dev/block/mmcblk0p9  [      16.00 MiB] 
  /dev/block/ram2       [       8.00 MiB] 
  /dev/block/mmcblk0p10 [     654.00 MiB] 
  /dev/block/mmcblk0p2  [       3.50 MiB] 
  /dev/block/ram3       [       8.00 MiB] 
  /dev/block/mmcblk0p11 [     432.00 MiB] 
  /dev/block/mmcblk0p3  [      20.00 MiB] 
  /dev/block/ram4       [       8.00 MiB] 
  /dev/block/mmcblk0p12 [      13.54 GiB] 
  /dev/block/mmcblk0p4  [       8.00 MiB] 
  /dev/block/ram5       [       8.00 MiB] 
  /dev/block/mmcblk0p5  [       4.00 MiB] 
  /dev/block/ram6       [       8.00 MiB] 
  /dev/block/mmcblk0p6  [       4.00 MiB] 
  /dev/block/ram7       [       8.00 MiB] 
  /dev/block/mmcblk0p7  [       8.00 MiB] 
  /dev/block/ram8       [       8.00 MiB] 
  /dev/block/ram9       [       8.00 MiB] 
  /dev/block/ram10      [       8.00 MiB] 
  /dev/block/ram11      [       8.00 MiB] 
  /dev/block/ram12      [       8.00 MiB] 
  /dev/block/ram13      [       8.00 MiB] 
  /dev/block/ram14      [       8.00 MiB] 
  /dev/block/ram15      [       8.00 MiB] 
  0 disks
  25 partitions
  0 LVM physical volume whole disks
  2 LVM physical volumes
root@maguro:/data # lvm vgcreate vg1 /dev/block/ram0 /dev/block/ram1
  Volume group "vg1" successfully created
root@maguro:/data # lvm vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "vg1" using metadata type lvm2
root@maguro:/data # lvm vgdisplay vg1                                          
  --- Volume group ---
  VG Name               vg1
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               8.00 MiB
  PE Size               4.00 MiB
  Total PE              2
  Alloc PE / Size       0 / 0   
  Free  PE / Size       2 / 8.00 MiB
  VG UUID               v2XaKu-OajI-lgF8-Vkcu-0NJd-ZyNg-wNdXnR
root@maguro:/data # lvm lvcreate -n test --size 1m vg1                         
  Rounding up size to full physical extent 4.00 MiB
  Logical volume "test" created
root@maguro:/data # ls /dev/vg1/test                                           
/dev/vg1/test
root@maguro:/data # mkfs.
mkfs.ext2  mkfs.minix mkfs.vfat  
root@maguro:/data # mkfs.ext2 /dev/vg1/test                                    
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
1024 inodes, 4096 blocks
204 blocks (5%) reserved for the super user
First data block=1
Maximum filesystem blocks=262144
1 block groups
8192 blocks per group, 8192 fragments per group
1024 inodes per group
root@maguro:/data # mkdir mnttest  
root@maguro:/data # mount -t ext2 /dev/vg1/test mnttest/                     
root@maguro:/data # ls mnttest/ 
lost+found
root@maguro:/data # lvm lvdisplay
  --- Logical volume ---
  LV Path                /dev/vg1/test
  LV Name                test
  VG Name                vg1
  LV UUID                U76Oyx-oryG-w95y-3cDD-Tf1r-MNdM-PsT6t5
  LV Write Access        read/write
  LV Creation host, time localhost, 2014-11-13 19:24:45 +0000
  LV Status              available
  # open                 1
  LV Size                4.00 MiB
  Current LE             1
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
root@maguro:/data # umount mnttest/                                            
root@maguro:/data # lvm lvremove /dev/vg1/test 
root@maguro:/data # lvm lvcreate -L 4M -m1 --mirrorlog core -n mirrorlv vg1 /dev/block/ram0 /dev/block/ram1
  Logical volume "mirrorlv" created
root@maguro:/data # mkfs.ext2 /dev/vg1/mirrorlv                              
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
1024 inodes, 4096 blocks
204 blocks (5%) reserved for the super user
First data block=1
Maximum filesystem blocks=262144
1 block groups
8192 blocks per group, 8192 fragments per group
1024 inodes per group
root@maguro:/data # mount -t ext2 /dev/vg1/mirrorlv mnttest/                 
root@maguro:/data # ls mnttest/                                                
lost+found
root@maguro:/data # cd mnttest/                                                
root@maguro:/data/mnttest # mkdir hello
root@maguro:/data/mnttest # ls
hello
lost+found

The only weird thing is that ram0 and ram1 seem to stay different even after I have accessed the mirrored logical volume. I must have missed some detail here. Don't know if this issue is ram-specific though.

root@maguro:/data # sha1sum /dev/block/ram1                                    
ed82c9f422e5b3c39c0f6e60fbd6175b9c3b5826  /dev/block/ram1
root@maguro:/data # sha1sum /dev/block/ram0                                    
f156266ca7858cd01389f4473d3b866d5033f780  /dev/block/ram0

wenluhu commented 10 years ago

I guess it makes sens the checksums of ram0 and ram1 differ. The headers are slightly different at least.

root@maguro:/data # head /dev/block/ram0
LABELONE
        LVM2 001lAMucJSxXatugzRfwevkrdvArd4UemeZ���P� LVM2 x[5A%r0N*><7I�]zvg1 {
id = "v2XaKu-OajI-lgF8-Vkcu-0NJd-ZyNg-wNdXnR"
seqno = 1
format = "lvm2" # informational
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192
max_lv = 0
max_pv = 0
metadata_copies = 0
root@maguro:/data # head /dev/block/ram1                                       
LABELONEWF# LVM2 001x8YrUXrQQtzE3ukMcg03giPY7K01aVmx���P� LVM2 x[5A%r0N*><7I�]zvg1 {
id = "v2XaKu-OajI-lgF8-Vkcu-0NJd-ZyNg-wNdXnR"
seqno = 1
format = "lvm2" # informational
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192
max_lv = 0
max_pv = 0
metadata_copies = 0

So I tried another way to see if ram0 is properly mirrored by ram1. I believe this proves lvm mirroring is working! :) I'm closing this issure.

root@maguro:/data # cat /dev/block/ram1 | grep Wenlu                           
1|root@maguro:/data # cat /dev/block/ram0 | grep Wenlu                         
1|root@maguro:/data # lvm lvcreate -L 4M -m1 --mirrorlog core -n mirrorlv vg1 /dev/block/ram0 /dev/block/ram1
  Logical volume "mirrorlv" created
root@maguro:/data # mkfs.ext2 /dev/vg1/mirrorlv
...
root@maguro:/data # mount -t ext2 /dev/vg1/mirrorlv mnttest/
root@maguro:/data # cd mnttest/                                                
root@maguro:/data/mnttest # ls
lost+found
root@maguro:/data/mnttest # mkdir Wenlu
root@maguro:/data/mnttest # cat /dev/block/ram0 | grep Wenlu                   
Binary file (standard input) matches
root@maguro:/data/mnttest # cat /dev/block/ram1 | grep Wenlu                   
Binary file (standard input) matches

wenluhu / gammaray-android

Compile/run device mapper user space utils on Android #1