Investigate gray-nbd-queuer

[bamos]

• [x] On server
• [x] Test disconnecting/adding nbd

[bamos]

@theonewolf

https://github.com/cmusatyalab/gammaray/issues/12 is a related issue in gammaray. I've tried modifying the following instructions from your readme and will poke around more later.

I created a disk from the 'Example Creation of gammaray-Supported Disk Layout' section. Then from the pipeline section:

./gray-crawler disk.raw disk.bson
mkfifo disk.fifo
./gray-ndb-queuer disk.bson disk.fifo 4 1>queuer.log 2>queuer.error.log &
./gray-inferencer disk.bson 4 disk_test_instance &
./test/nbd-test disk disk.raw 0.0.0.0 202020 y 2>disk.fifo

This outputs:

gammaray/bin [master*] » ./gray-crawler disk.raw disk.bson
Raw Disk Crawler -- By: Wolfgang Richter <wolf@cs.cmu.edu>
Analyzing Disk: disk.raw

Probing for mbr... 

Analyzing Boot Sector
Disk Signature [optional]: 0x77e6d609
Position 444 [0x0000]: 0x0000
Verifying MBR Signature [0x55 0xaa]: 0x55 0xaa

Checking partition table entry 0.
Status [0x80 bootable, 0x00 non-bootable]: 0x00
Partition Type: 0x83
Partition Type: Linux
Start Head: 0x00
Start Sector: 0x01
Start Cylinder: 0x010
End Head: 0x03
End Sector: 0x20
End Cylinder: 0x3ff
First Sector LBA: 0x00000800
Number of Sectors: 0x00989000 (4.768 GiB)

Checking partition table entry 1.
Status [0x80 bootable, 0x00 non-bootable]: 0x00
Partition Type: 0x00
Partition Type: Empty

Checking partition table entry 2.
Status [0x80 bootable, 0x00 non-bootable]: 0x00
Partition Type: 0x00
Partition Type: Empty

Checking partition table entry 3.
Status [0x80 bootable, 0x00 non-bootable]: 0x00
Partition Type: 0x00
Partition Type: Empty
found mbr partition table!

Probing for ext4... found ext4 file system!
gammaray/bin [master*] » mkfifo disk.fifo
mkfifo: cannot create fifo ‘disk.fifo’: File exists
gammaray/bin [master*] » ./gray-ndb-queuer disk.bson disk.fifo 4 1>queuer.log 2>queuer.error.log &
[1] 17033
gammaray/bin [master*] » ./gray-inferencer disk.bson 4 disk_test_instance &
[2] 17065
gammaray Inference Engine -- By: Wolfgang Richter <wolf@cs.cmu.edu>                             
Current libhiredis version is 0.11.0
disk_test_instance: loading index: disk.bson

-- Deserializing a mbr record --
-- Deserializing a partition record --
-- Deserializing a fs record --
gammaray/bin [master*] » ./test/nbd-test disk disk.raw 0.0.0.0 202020 y 2>disk.fifo-- Deserializing a bitarray record --
-- Deserialized 39 bgd's --
-- Deserialized 4 file's -

Next, I started an nbd client and tried modifying a file named test-file that I created before the above sequence.

/tmp » _ nbd-client derecho.elijah.cs.cmu.edu 202020 /dev/nd0
Negotiation: ..size = 5120MB
bs=1024, sz=5368709120 bytes
/tmp » _ kpartx -av /dev/nd0
add map nd0p1 (254:1): 0 9998336 linear /dev/nd0 2048
/tmp » _ mount /dev/mapper/nd0p1 /tmp/nd0
/tmp » cd nd0
/tmp/nd0 » ls
lost+found  test-file
/tmp/nd0 » vim test-file
/tmp/nd0 » ls
lost+found  test-file
/tmp/nd0 »

However, nothing shows up in the output on the gammaray side, so maybe directing the nbd-test output to the named pipe isn't correct.

The entire stream of reading/writing to a file looks like:

gammaray/bin [master*] » ./test/nbd-test disk disk.raw 0.0.0.0 202020 y                      <<<
nbd-test program by: Wolfgang Richter <wolf@cs.cmu.edu>
++++++  [1415968271365597] write size: 8192
    write[1]: 8192 cumulative bytes (8.000 KiB)
++++    [1415968271412301] write size: 4096
    write[2]: 12288 cumulative bytes (12.000 KiB)
++++    [1415968271462721] write size: 4096
    write[3]: 16384 cumulative bytes (16.000 KiB)
got flush.
+++ [1415968271941448] write size: 4096
    write[4]: 20480 cumulative bytes (20.000 KiB)
got flush.
+++ [1415968272363156] write size: 4096
    write[5]: 24576 cumulative bytes (24.000 KiB)
+++ [1415968272441250] write size: 4096
    write[6]: 28672 cumulative bytes (28.000 KiB)
got flush.
+++ [1415968278704499] write size: 4096
    write[7]: 32768 cumulative bytes (32.000 KiB)
+++ [1415968278806193] write size: 4096
    write[8]: 36864 cumulative bytes (36.000 KiB)
+++++++++++++++ [1415968279069271] write size: 20480
    write[9]: 57344 cumulative bytes (56.000 KiB)
got flush.
+++ [1415968279489009] write size: 4096
    write[10]: 61440 cumulative bytes (60.000 KiB)
got flush

Should I run the nbd-queuer-test instead?

[theonewolf]

Oh yeah, pipes aren't used anymore. Sorry for the confusion.

So I was transitioning gammaray to nbd, but the transition wasn't complete (I want an in-memory tunable buffer in nbd-test, then I'd consider it a full replacement for gray-nbd-queuer).

We can go over things later today in person though.

On Fri, Nov 14, 2014 at 7:33 AM, Brandon Amos notifications@github.com wrote:

@theonewolf https://github.com/theonewolf

cmusatyalab#12 https://github.com/cmusatyalab/gammaray/issues/12 is a related issue in gammaray. I've tried modifying the following instructions from your readme and will poke around more later.

I created a disk from the 'Example Creation of gammaray-Supported Disk Layout' section. Then from the pipeline section:

./gray-crawler disk.raw disk.bson mkfifo disk.fifo ./gray-ndb-queuer disk.bson disk.fifo 4 1>queuer.log 2>queuer.error.log & ./gray-inferencer disk.bson 4 disk_test_instance & ./test/nbd-test disk disk.raw 0.0.0.0 202020 y 2>disk.fifo

This outputs:

gammaray/bin [master*] » ./gray-crawler disk.raw disk.bson Raw Disk Crawler -- By: Wolfgang Richter wolf@cs.cmu.edu Analyzing Disk: disk.raw

Probing for mbr...

Analyzing Boot Sector Disk Signature [optional]: 0x77e6d609 Position 444 [0x0000]: 0x0000 Verifying MBR Signature [0x55 0xaa]: 0x55 0xaa

Checking partition table entry 0. Status [0x80 bootable, 0x00 non-bootable]: 0x00 Partition Type: 0x83 Partition Type: Linux Start Head: 0x00 Start Sector: 0x01 Start Cylinder: 0x010 End Head: 0x03 End Sector: 0x20 End Cylinder: 0x3ff First Sector LBA: 0x00000800 Number of Sectors: 0x00989000 (4.768 GiB)

Checking partition table entry 1. Status [0x80 bootable, 0x00 non-bootable]: 0x00 Partition Type: 0x00 Partition Type: Empty

Checking partition table entry 2. Status [0x80 bootable, 0x00 non-bootable]: 0x00 Partition Type: 0x00 Partition Type: Empty

Checking partition table entry 3. Status [0x80 bootable, 0x00 non-bootable]: 0x00 Partition Type: 0x00 Partition Type: Empty found mbr partition table!

Probing for ext4... found ext4 file system! gammaray/bin [master] » mkfifo disk.fifo mkfifo: cannot create fifo ‘disk.fifo’: File exists gammaray/bin [master] » ./gray-ndb-queuer disk.bson disk.fifo 4 1>queuer.log 2>queuer.error.log & [1] 17033 gammaray/bin [master*] » ./gray-inferencer disk.bson 4 disk_test_instance & [2] 17065 gammaray Inference Engine -- By: Wolfgang Richter wolf@cs.cmu.edu Current libhiredis version is 0.11.0 disk_test_instance: loading index: disk.bson

-- Deserializing a mbr record -- -- Deserializing a partition record -- -- Deserializing a fs record -- gammaray/bin [master*] » ./test/nbd-test disk disk.raw 0.0.0.0 202020 y 2>disk.fifo-- Deserializing a bitarray record -- -- Deserialized 39 bgd's -- -- Deserialized 4 file's -

Next, I started an nbd client and tried modifying a file named test-file that I created before the above sequence.

/tmp » nbd-client derecho.elijah.cs.cmu.edu 202020 /dev/nd0 Negotiation: ..size = 5120MB bs=1024, sz=5368709120 bytes /tmp » kpartx -av /dev/nd0 add map nd0p1 (254:1): 0 9998336 linear /dev/nd0 2048 /tmp » _ mount /dev/mapper/nd0p1 /tmp/nd0 /tmp » cd nd0 /tmp/nd0 » ls lost+found test-file /tmp/nd0 » vim test-file /tmp/nd0 » ls lost+found test-file /tmp/nd0 »

However, nothing shows up in the output on the gammaray side, so maybe directing the nbd-test output to the named pipe isn't correct.

The entire stream of reading/writing to a file looks like:

gammaray/bin [master*] » ./test/nbd-test disk disk.raw 0.0.0.0 202020 y <<< nbd-test program by: Wolfgang Richter wolf@cs.cmu.edu ++++++ [1415968271365597] write size: 8192 write[1]: 8192 cumulative bytes (8.000 KiB) ++++ [1415968271412301] write size: 4096 write[2]: 12288 cumulative bytes (12.000 KiB) ++++ [1415968271462721] write size: 4096 write[3]: 16384 cumulative bytes (16.000 KiB) got flush. +++ [1415968271941448] write size: 4096 write[4]: 20480 cumulative bytes (20.000 KiB) got flush. +++ [1415968272363156] write size: 4096 write[5]: 24576 cumulative bytes (24.000 KiB) +++ [1415968272441250] write size: 4096 write[6]: 28672 cumulative bytes (28.000 KiB) got flush. +++ [1415968278704499] write size: 4096 write[7]: 32768 cumulative bytes (32.000 KiB) +++ [1415968278806193] write size: 4096 write[8]: 36864 cumulative bytes (36.000 KiB) +++++++++++++++ [1415968279069271] write size: 20480 write[9]: 57344 cumulative bytes (56.000 KiB) got flush. +++ [1415968279489009] write size: 4096 write[10]: 61440 cumulative bytes (60.000 KiB) got flush

Should I run the nbd-queuer-test instead?

— Reply to this email directly or view it on GitHub https://github.com/wenluhu/gammaray-android/issues/20#issuecomment-63057555 .

Wolf

[bamos]

I'll look into:

• [ ] Can the nbd be mounted on Android without kpartx?

[theonewolf]

This should be possible, I'd assume once the block device appears it will get auto-probed by the kernel.

[theonewolf]

Getting nbd-queuer-test ready to duplicate writes to both Redis and a file:

• [x] Create an init function based on nbd_init_redis that also opens and sets an fd
  • https://github.com/wenluhu/gammaray-android/blob/80a478a55f8841fb43b5a5f69dd2882ab65937ea/src/nbd/nbd.c#L802
  • I'd call this function nbd_init_both
• [x] Hack __handle_write to send a write to both Redis and the fd/file
  • Comment out the if (or put suitable logic here)
  • Comment out the early return (or have suitable logic)
  • https://github.com/wenluhu/gammaray-android/blob/80a478a55f8841fb43b5a5f69dd2882ab65937ea/src/nbd/nbd.c#L448

[theonewolf]

This might be helpful, when testing, the nbd-test program is setup to use a backing file, not Redis.

Took me awhile to remember how to hook things together using the "newstyle" NBD protocol.

At the "nbd server" you do something like this:

wolf@fog ~/g/test> ./nbd-test test /tmp/fake-drive.raw 0.0.0.0 10809 n

at the "nbd client" you do this (notice the "share" name -N is matching---an important bit):

➜ ~>sudo nbd-client -N test fog.elijah.cs.cmu.edu /dev/nbd0
Negotiation: ..size = 8192MB
bs=1024, sz=8589934592 bytes

The idea with your hack is to have a nbd-test-both program/version that will do both Redis + file backing mode.

[theonewolf]

Using fdisk I created a new MBR/partition table which resulted in this log at the nbd server:

got flush.
got flush.
+   [1416005606230256] write size: 1024
    write[1]: 1024 cumulative bytes (1024 B)
got flush.
got disconnect.

I further reconnected and then created an ext4 file system as the first partition (this is a well-tested mode for gammaray):

https://gist.githubusercontent.com/theonewolf/22304090a6feab570bcb/raw/f587fcd2c9921c6e34a9b297066da07535a5ae9a/nbd.log

Then I, just to make sure, stress tested reading and writing by putting in a known random file of size 5 gigabytes (big enough to stress large file support at all layers). I wrote it out and then read it back verifying the bytes. I also checked the throughput numbers and found that my nbd server code is able to push data at close to network-bandwidth rates (essentially we can probably saturate a 1 Gbps link):

https://gist.github.com/theonewolf/1ce6f6030842acd69cdd

Note that the write rate isn't saturating (theoretical maximum of 125 MBps power of 10), that's because the nbd kernel code does periodic "flush" operations which slows down writing (0 bandwidth for awhile).

[bamos]

Hi @theonewolf and @wenluhu - I've made modifications to nbd-queuer-test and have created some scripts for the sever side of the demo here: https://github.com/wenluhu/gammaray-android/tree/demo/demo

These run on the server and will run the gammaray server code and create a raw disk file with ext4 and a single file. One assumption is that redis is running.

The test-client.sh code runs on the server (for now) and will connect to the nbd export and modify the existing file by using the shell's append operator >>. Take a look at the log here: https://gist.github.com/anonymous/fcce62c96cfe87114faa I believe the inferencer log is showing that this modification is correctly logged.

Three sub-issues for where I'm at now are, can you review them when possible, Wolf?

• [ ] The inferencer isn't showing the new file being created. Is this scoped enough for me to help with code, or is this something you want to do?
• [ ] Do you expect all of the empty writes?
• [ ] The gray-fs mount doesn't seem to show any of the block device's contents. Did I do something wrong in the server initialization scripts? This only relies on the block device, right?
• [ ] I tried to run the nbd client code on the phone to see if I can mount the partition and see a file modification. However, I don't know how to extract the partition from the block device without kpartx. I'm using the following from a root Android shell.

root@maguro:/data/gray # /data/nbdroid/nbd-client 192.168.1.2 30000 /dev/nbd0
Negotiation: ..size = 5120MB
bs=1024, sz=5368709120 bytes

root@maguro:/data/gray # ls /dev/nbd0*                                         
/dev/nbd0

root@maguro:/data/gray # mkdir -p test-mount

1|root@maguro:/data/gray # mount -r -w -t ext4 /dev/nbd0 test-mount            
mount: Invalid argument

Am I overlooking something here?

• [x] Would it be hard to modify the nbd queuer to export the partition rather than the block device?
• [x] I could also try to compile kpartx or a similar tool for Android.

[bamos]

The partition tables offset will be the same for the disk. Using this reference, the offset will always be 2024*512 = 103628.

gammaray-android/demo [demo] » sudo fdisk -lu disk.raw

Disk disk.raw: 5 GiB, 5368709120 bytes, 10485760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xbd16c59a

Device     Boot Start      End Sectors  Size Id Type
disk.raw1        2048 10000383 9998336  4.8G 83 Linux

I've installed busybox (which includes nbd-client and mknod!) for losetup. Unfortunately, the loop device doesn't seem to be correctly mounted with the offset.

root@maguro:/data/nbdroid # busybox nbd-client 192.168.1.2 30000 /dev/block/nbd0
root@maguro:/data/nbdroid # busybox mknod /dev/gray-loop b 7 8
root@maguro:/data/nbdroid # busybox losetup -o 1036288 /dev/gray-loop /dev/block/nbd0
root@maguro:/data/nbdroid # mkdir p-mount
root@maguro:/data/nbdroid # mount -r -w -t ext /dev/gray-loop p-mount
130|root@maguro:/data/nbdroid # dd if=/dev/gray-loop bs=100 count=1            
1+0 records in
1+0 records out
100 bytes transferred in 0.092 secs (1086 bytes/sec

The following snippet confirms that the offset option of losetup is causing this! I can read from the loop device when an offset isn't provided.

root@maguro:/data/nbdroid # busybox losetup /dev/block/loop0 /dev/block/nbd0                     <
root@maguro:/data/nbdroid # dd if=/dev/block/loop0 bs=100 count=1
ú¸м°¸؎Àû¾|¿¹ó¤ê!¾¾8u
                    Æþþuóë´°»|²tLÍê|ëþ1+0 records in
1+0 records out
100 bytes transferred in 0.026 secs (3846 bytes/sec)
root@maguro:/data/nbdroid # busybox losetup -d /dev/block/loop0                
root@maguro:/data/nbdroid # busybox losetup -o 100 /dev/block/loop0 /dev/block>
root@maguro:/data/nbdroid # dd if=/dev/block/loop0 bs=100 count=1              
1+0 records in
1+0 records out
100 bytes transferred in 0.001 secs (100000 bytes/sec

• [ ] This possibly might work better with CyanogenMod, but I'm doubtful. Otherwise, I'm not sure what else to try.

[bamos]

Hm, I also can't seem to get mount provided by busybox or stock android to work:

255|root@maguro:/data/nbdroid # busybox mount -o loop,offset=1036288 -t ext4 /dev/block/nbd0 /dev/block/loop3
mount: mounting /dev/block/nbd0 on /dev/block/loop3 failed: Invalid argument
``

[theonewolf]

nbd-queuer could be customized to only export the partition, it'd take some hacking. I'm not sure what is causing auto-probing of attached block devices on my system, but I guess it isn't auto running on Android.

Using the partition table you'd just need to offset every read and write by some constant.

Perhaps the best thing here would be to add a global constant offset to the queuer, and set it on the command line with the offset to partition derived from any tool you want even by hand.

(1) I'll re-enable/debug the current state of new file detection starting from 4 or 5 PM.

(2) dunno about empty writes right now. Would have to observe.

(3) gray-fs works from BSON loaded by gray-inferencer into Redis and pulls data from the virtual disk file.

Writing this before a long drive, hope something is helpful.

Sent from my iPhone

On Nov 15, 2014, at 5:03 AM, Brandon Amos notifications@github.com wrote:

Hi @theonewolf and @wenluhu - I've made modifications to nbd-queuer-test and have created some scripts for the sever side of the demo here: https://github.com/wenluhu/gammaray-android/tree/demo/demo

These run on the server and will run the gammaray server code and create a raw disk file with ext4 and a single file. One assumption is that redis is running.

The test-client.sh code runs on the server (for now) and will connect to the nbd export and modify the existing file by using the shell's append operator >>. Take a look at the log here: https://gist.github.com/anonymous/fcce62c96cfe87114faa I believe the inferencer log is showing that this modification is correctly logged.

Three sub-issues for where I'm at now are, can you review them when possible, Wolf?

The inferencer isn't showing the new file being created. Is this scoped enough for me to help with code, or is this something you want to do? Do you expect all of the empty writes? The gray-fs mount doesn't seem to show any of the block device's contents. Did I do something wrong in the server initialization scripts? This only relies on the block device, right? I tried to run the nbd client code on the phone to see if I can mount the partition and see a file modification. However, I don't know how to extract the partition from the block device without kpartx. I'm using the following from a root Android shell. root@maguro:/data/gray # /data/nbdroid/nbd-client 192.168.1.2 30000 /dev/nbd0 Negotiation: ..size = 5120MB bs=1024, sz=5368709120 bytes

root@maguro:/data/gray # ls /dev/nbd0*
/dev/nbd0

root@maguro:/data/gray # mkdir -p test-mount

1|root@maguro:/data/gray # mount -r -w -t ext4 /dev/nbd0 test-mount
mount: Invalid argument Am I overlooking something here?

Would it be hard to modify the nbd queuer to export the partition rather than the block device? I could also try to compile kpartx or a similar tool for Android. — Reply to this email directly or view it on GitHub.

[bamos]

I flashed my phone with CyanogenMod, which includes busybox by default. However, I couldn't even get the nbd-client working, so I'm now reverting to stock Android...

Hacky solution I'm trying afterwards:

• GammaRay exports nbd of entire block device on tchaikovsky (home laptop)
• tchaikovsky uses kpartx to mount partition to /dev/mapper/loopXp1.
• tchaikovsky exports /dev/mapper/loopXp1 as an nbd share
• Android mounts the partition from the nbd share

[bamos]

Fantastic, the following works to export the partition of an nbd share! This is on my Linux computer, will hopefully transfer to Android when I reflash my device...

Terminal 1:

~/repos » _ nbd-client localhost 30000 /dev/nbd0
Negotiation: ..size = 5120MB
bs=1024, sz=5368709120 bytes
~/repos » _ kpartx -av /dev/nbd0
add map nbd0p1 (254:1): 0 9998336 linear /dev/nbd0 2048
~/repos » _ qemu-nbd -p 30001 /dev/mapper/nbd0p1

Terminal 2 (hopefully Android)

/tmp » _ nbd-client localhost 30001 /dev/nbd1
Negotiation: ..size = 4882MB
bs=1024, sz=5119148032 bytes
/tmp » _ mount /dev/nbd1 p-mount-test
/tmp » cd p-mount-test
/tmp/p-mount-test » ls
lost+found  testfile

[theonewolf]

Sweet mother of God. nbd within nbd.

The spirit of working demo is with you. At all costs!

Following this line of thinking, as a fall back plan to mitigate disaster from your mentor:

Setup a script which runs gray-crawler periodically, then runs redis flushall, then gray-inference-engine component to load the generated BSON into Redis, then gray-fs to mount the file system again. Of course, terminate gray-inference-engine and unmount the FUSE file system and terminate gray-fs on every loop if they are running as well.

You might get away with leaving gray-fs running all the time, and it might not crash. It reads freshly from Redis (or it should) every time you try to traverse directories. Just don't read in the middle of a gray-inference-engine load :p

This will simulate live introspection of even file creation etc.

You could, of course, try mounting directly with regular tools/the Linux kernel---but you run the danger of potentially causing a kernel panic of your demo system (thus, using gammaray tools is safer).

On Sun, Nov 16, 2014 at 6:13 PM, Brandon Amos notifications@github.com wrote:

Fantastic, the following works to export the partition of an nbd share! This is on my Linux computer, will hopefully transfer to Android when I reflash my device...

Terminal 1:

~/repos » nbd-client localhost 30000 /dev/nbd0 Negotiation: ..size = 5120MB bs=1024, sz=5368709120 bytes ~/repos » kpartx -av /dev/nbd0 add map nbd0p1 (254:1): 0 9998336 linear /dev/nbd0 2048 ~/repos » _ qemu-nbd -p 30001 /dev/mapper/nbd0p1

Terminal 2 (hopefully Android)

/tmp » nbd-client localhost 30001 /dev/nbd1 Negotiation: ..size = 4882MB bs=1024, sz=5119148032 bytes /tmp » mount /dev/nbd1 p-mount-test /tmp » cd p-mount-test /tmp/p-mount-test » ls lost+found testfile

— Reply to this email directly or view it on GitHub https://github.com/wenluhu/gammaray-android/issues/20#issuecomment-63244455 .

Wolf