search

wenluhu / gammaray-android

Distributed Streaming Virtual Machine Introspection
Apache License 2.0
6 stars 1 forks source link

Error running gray-crawler on Android #17

Closed bamos closed 10 years ago

bamos commented 10 years ago

GPT is working well, filesystem checks aren't.

screen shot 2014-10-14 at 2 32 02 pm

screen shot 2014-10-14 at 2 30 28 pm

bamos commented 10 years ago

From @theonewolf:

can we check and make sure /data is not encrypted? not sure on the best way of checking within Android

maybe, from https://source.android.com/devices/tech/encryption/android_crypto_implementation.html:

The /data filesystem must be on a device that presents a block device interface. eMMC is used in the first devices. This is because the encryption is done by the dm-crypt layer in the kernel, which works at the block device layer.

Need a more definite answer on this, I'm running Android 4.3

bamos commented 10 years ago

Return of fseeko is -1:

2014-10-21-084855_504x118_scrot

2014-10-21-084906_316x69_scrot

bamos commented 10 years ago

Further debug info on this: offsets and mount options.

2014-10-21-090653_309x65_scrot

2014-10-21-090703_631x69_scrot

2014-10-21-090723_568x446_scrot

bamos commented 10 years ago

Hi @theonewolf - does any of this debug info throw any flags for you? Perhaps I'm incorrectly setting pte->pt_off?

bamos commented 10 years ago

Also @theonewolf - I tried extracting Android's block device as a file with adb shell su -c 'dd if=/dev/block/mmcblk0' > mmcblk0.raw, and it extracted an 11GB image. However, gammaray-crawler is performing different on it than on the device.

bamos commented 10 years ago

The file appears to have the correct GPT, though... gammaray-crawler is able to read the GPT, but doesn't recognize any of the filesystems.

2014-10-21-091433_558x448_scrot

theonewolf commented 10 years ago

Right, sounds like we should have a pair-debug session maybe in the near future.

I'll ping you with a good time.

On Tue, Oct 21, 2014 at 9:14 AM, Brandon Amos notifications@github.com wrote:

The file appears to have the correct headers, though...

[image: 2014-10-21-091433_558x448_scrot] https://cloud.githubusercontent.com/assets/707462/4718815/347b18ac-5924-11e4-8f10-2f686f391809.png

— Reply to this email directly or view it on GitHub https://github.com/wenluhu/gammaray-android/issues/17#issuecomment-59924939 .

Wolf

bamos commented 10 years ago

Sounds good, thanks!

bamos commented 10 years ago

From root@maguro:/dev/block/platform/omap/omap_hsmmc.0/by-name # ll

Block Device Number Alias Filesystem Mounted
1 xloader
2 sbl
3 efs ext4 /factory
4 param
5 misc
6 dgs
7 boot
8 recovery
9 radio
10 system
11 cache
12 userdata ext4 /data
13 metadata

2014-10-24-164906_556x695_scrot

bamos commented 10 years ago
bamos commented 10 years ago
theonewolf commented 10 years ago

Looked into the superblock.

This is what the tune2fs thinks of it:

wolf@fog ~/Dropbox> tune2fs -l fake_ext4.raw 
tune2fs 1.42.9 (4-Feb-2014)
Filesystem volume name:   <none>
Last mounted on:          /data
Filesystem UUID:          57f8f4bc-abf4-655f-bf67-946fc0f9f25b
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode filetype needs_recovery extent sparse_super large_file uninit_bg
Filesystem flags:         (none)
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Remount read-only
Filesystem OS type:       Linux
Inode count:              887696
Block count:              3549691
Reserved block count:     0
Free blocks:              3056181
Free inodes:              851573
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      871
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8144
Inode blocks per group:   509
Last mount time:          Fri Oct 24 15:54:28 2014
Last write time:          Fri Oct 24 15:54:28 2014
Mount count:              105
Maximum mount count:      -1
Last checked:             Wed Dec 31 19:00:00 1969
Check interval:           0 (<none>)
Lifetime writes:          21 TB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     7168
Desired extra isize:      7168
Journal inode:            8
First orphan inode:       578975
Default directory hash:   tea
Journal backup:           inode blocks

I created a "fake" home for it via:

truncate -s 4096 fake_ext4.raw
dd if=fs_info_failed.raw of=fake_ext4.raw seek=1 bs=1024 count=1

Basically, the superblock is supposed to appear 1024 bytes in, and can take up a maximum of 1024 bytes itself. So this is a "hack" to make legacy tools "believe" it is a good ext4 file system (albeit only up until the first superblock).

Note: I had to make it slightly bigger to make the tool happy. I think these tools try to read more than 2048 bytes, although I am not 100% sure (didn't look deeply into it, 4096 bytes was good enough to fake this).

theonewolf commented 10 years ago

I believe this block_count is the same as what gammaray is computing.

I thought that maybe one of the ext4 features that was turned off on the Android, but present on other systems by default, might be messing up gammaray.

I tried creating a fake ext4 file system with all those features disabled, but gammaray didn't choke on that on my host system (granted, it was an empty file system).

For the record, I was disabling these features:

^dir_index,^flex_bg,^huge_file,^dir_nlink,^extra_isize
theonewolf commented 10 years ago

Here is an example of a "good" superblock that gammaray has no trouble working with, in case you can see something I'm not @bamos:

wolf@fog ~/Dropbox> cat cloud.log 
tune2fs 1.42.9 (4-Feb-2014)
Filesystem volume name:   cloudimg-rootfs
Last mounted on:          /
Filesystem UUID:          420f8b1e-a8d8-4bcb-b650-8331903587eb
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              147456
Block count:              576256
Reserved block count:     26653
Free blocks:              366619
Free inodes:              89374
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      87
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Mon May 19 04:39:15 2014
Last mount time:          Sun Oct 26 03:25:06 2014
Last write time:          Mon May 19 04:42:59 2014
Mount count:              3
Maximum mount count:      -1
Last checked:             Mon May 19 04:39:15 2014
Check interval:           0 (<none>)
Lifetime writes:          48 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      373855d7-1d2e-475c-8d66-6ea3d505aa52
Journal backup:           inode blocks
theonewolf commented 10 years ago

@bamos also, not sure what torture you do to these phones, but 21 TB of writes? :laughing:

Wonder if that is some bug.

bamos commented 10 years ago

Haha! I have no idea where the 21 TB of writes came from.

theonewolf commented 10 years ago

Try diff'ing with your brain the two tune2fs outputs and see if anything looks suspicious that we can test. I tried the features, might try again to make sure I'm not crazy. Didn't read about all the features, but it could potentially still be one of them (and an empty file system doesn't mess up gammaray).

theonewolf commented 10 years ago

@bamos good idea on task lists, so:

bamos commented 10 years ago

dd from mmcblk0 to nbd0 (at derecho:~bamos/data/gray/gray-nbd-all successfully finished after 42 hours:

2014-10-27-185615_477x59_scrot

theonewolf commented 10 years ago

42 hours seems excessive. Did it take this long with the other method you tried? Or has nbd always been the method?

On Mon, Oct 27, 2014 at 6:57 PM, Brandon Amos notifications@github.com wrote:

dd from mmcblk0 to nbd0 (at derecho:~bamos/data/gray/gray-nbd-all successfully finished after 42 hours:

[image: 2014-10-27-185615_477x59_scrot] https://cloud.githubusercontent.com/assets/707462/4801047/7eb25b08-5e2c-11e4-85cb-917ef382ad37.png

— Reply to this email directly or view it on GitHub https://github.com/wenluhu/gammaray-android/issues/17#issuecomment-60684493 .

Wolf

bamos commented 10 years ago

Hm, I didn't time the other method.

The crawler's acting slightly different on this version:

~/repos/gammaray-gpt/bin/gray-crawler gray-nbd-all gray-nbd-all.bson && ~/repos/gammaray-gpt/bin/tools/bson-printer gray-nbd-all.bson > gray-nbd-all.bson-printed

Probing for ext4... ext4 superblock s_magic[0x946c] mismatch.
not found.

Probing for ntfs... NTFS probe failed.
not found.

Probing for ext4... ext4 superblock s_magic[0xffea] mismatch.
not found.

Probing for ntfs... NTFS probe failed.
not found.

Probing for ext4... ext4 superblock s_magic[0xef53] mismatch.
not found.

Probing for ntfs... NTFS probe failed.
not found.

Probing for ext4... ext4 superblock s_magic[0x0000] mismatch.
not found.

Probing for ntfs... NTFS probe failed.
not found.

Probing for ext4... ext4 superblock s_magic[0x0000] mismatch.
not found.

Probing for ntfs... NTFS probe failed.
not found.

Probing for ext4... ext4 superblock s_magic[0x0000] mismatch.
not found.

Probing for ntfs... NTFS probe failed.
not found.

Probing for ext4... ext4 superblock s_magic[0x0000] mismatch.
not found.

Probing for ntfs... NTFS probe failed.
not found.

Probing for ext4... ext4 superblock s_magic[0x0000] mismatch.
not found.

Probing for ntfs... NTFS probe failed.
not found.

Probing for ext4... ext4 superblock s_magic[0x6e31] mismatch.
not found.

Probing for ntfs... NTFS probe failed.
not found.

Probing for ext4... found ext4 file system!

Probing for ext4... found ext4 file system!

Probing for ext4... found ext4 file system!

Probing for ext4... ext4 superblock s_magic[0x0000] mismatch.
not found.

Probing for ntfs... NTFS probe failed.
not found.

The bson definitely has the /data files!

data/gray » grep '/data' -c gray-nbd-all.bson-printed
36274

The bson file's 17M:

data/gray » ll gray-nbd-all.bson
-rw-r--r-- 1 bamos bamos 17M Oct 29 14:14 gray-nbd-all.bson
theonewolf commented 10 years ago

Very cool, it is finding file systems this time I think (more than it did last time?).

I can debug this Sunday night/Monday as needed.

On Wed, Oct 29, 2014 at 2:17 PM, Brandon Amos notifications@github.com wrote:

Hm, I didn't time the other method.

The crawler's acting slightly different on this version:

~/repos/gammaray-gpt/bin/gray-crawler gray-nbd-all gray-nbd-all.bson && ~/repos/gammaray-gpt/bin/tools/bson-printer gray-nbd-all.bson > gray-nbd-all.bson-printed

Probing for ext4... ext4 superblock s_magic[0x946c] mismatch. not found.

Probing for ntfs... NTFS probe failed. not found.

Probing for ext4... ext4 superblock s_magic[0xffea] mismatch. not found.

Probing for ntfs... NTFS probe failed. not found.

Probing for ext4... ext4 superblock s_magic[0xef53] mismatch. not found.

Probing for ntfs... NTFS probe failed. not found.

Probing for ext4... ext4 superblock s_magic[0x0000] mismatch. not found.

Probing for ntfs... NTFS probe failed. not found.

Probing for ext4... ext4 superblock s_magic[0x0000] mismatch. not found.

Probing for ntfs... NTFS probe failed. not found.

Probing for ext4... ext4 superblock s_magic[0x0000] mismatch. not found.

Probing for ntfs... NTFS probe failed. not found.

Probing for ext4... ext4 superblock s_magic[0x0000] mismatch. not found.

Probing for ntfs... NTFS probe failed. not found.

Probing for ext4... ext4 superblock s_magic[0x0000] mismatch. not found.

Probing for ntfs... NTFS probe failed. not found.

Probing for ext4... ext4 superblock s_magic[0x6e31] mismatch. not found.

Probing for ntfs... NTFS probe failed. not found.

Probing for ext4... found ext4 file system!

Probing for ext4... found ext4 file system!

Probing for ext4... found ext4 file system!

Probing for ext4... ext4 superblock s_magic[0x0000] mismatch. not found.

Probing for ntfs... NTFS probe failed. not found.

The bson definitely has the /data files!

data/gray » grep '/data' -c gray-nbd-all.bson-printed 36274

The bson file's 17M:

data/gray » ll gray-nbd-all.bson -rw-r--r-- 1 bamos bamos 17M Oct 29 14:14 gray-nbd-all.bson

  • I'll add some simple traces to gray and find out where the x86 version diverges from the ARM version.

— Reply to this email directly or view it on GitHub https://github.com/wenluhu/gammaray-android/issues/17#issuecomment-60976759 .

Wolf

bamos commented 10 years ago

Block device sizes:

shell@maguro:/ $ cat /proc/partitions
major minor  #blocks  name

  31        0       1024 mtdblock0
 179        0   15388672 mmcblk0
 179        1        128 mmcblk0p1
 179        2       3584 mmcblk0p2
 179        3      20480 mmcblk0p3
 179        4       8192 mmcblk0p4
 179        5       4096 mmcblk0p5
 179        6       4096 mmcblk0p6
 179        7       8192 mmcblk0p7
 259        0      12224 mmcblk0p8
 259        1      16384 mmcblk0p9
 259        2     669696 mmcblk0p10
 259        3     442368 mmcblk0p11
 259        4   14198767 mmcblk0p12
 259        5         64 mmcblk0p13
 179       16        512 mmcblk0boot1
 179        8        512 mmcblk0boot0
bamos commented 10 years ago

@theonewolf - is /proc/partitions acceptable in place of running lsblk on Android?

bamos commented 10 years ago

I have tune2fs working on Android. Here's the output on the /data/ partition:

root@maguro:/data/gray # ./tune2fs -l /dev/block/mmcblk0p12
tune2fs 1.43-WIP (29-Aug-2014)
Filesystem volume name:   <none>
Last mounted on:          /data
Filesystem UUID:          57f8f4bc-abf4-655f-bf67-946fc0f9f25b
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode filetype needs_recovery extent sparse_super large_file uninit_bg
Filesystem flags:         unsigned_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Remount read-only
Filesystem OS type:       Linux
Inode count:              887696
Block count:              3549691
Reserved block count:     0
Free blocks:              3055759
Free inodes:              851521
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      871
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8144
Inode blocks per group:   509
Last mount time:          Mon Oct 27 18:59:36 2014
Last write time:          Mon Oct 27 18:59:36 2014
Mount count:              109
Maximum mount count:      -1
Last checked:             Wed Dec 31 19:00:00 1969
Check interval:           0 (<none>)
Lifetime writes:          108 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
First orphan inode:       578975
Default directory hash:   tea
Journal backup:           inode blocks
theonewolf commented 10 years ago

Great! I think what we'll do today is try to create a file system with the same settings, maybe tweaking from a default ext4 one by one until we hit something gammaray can't deal with.

theonewolf commented 10 years ago

Default ext4 options:

add map loop0p1 (253:2): 0 28397528 linear /dev/loop0 2048
tune2fs 1.42.12 (29-Aug-2014)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          16b12198-8928-4c10-a76b-3197ccba57c8
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              887696
Block count:              3549691
Reserved block count:     177484
Free blocks:              3452538
Free inodes:              887685
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      866
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8144
Inode blocks per group:   509
Flex block group size:    16
Filesystem created:       Tue Nov  4 13:37:29 2014
Last mount time:          n/a
Last write time:          Tue Nov  4 13:37:29 2014
Mount count:              0
Maximum mount count:      -1
Last checked:             Tue Nov  4 13:37:29 2014
Check interval:           0 (<none>)
Lifetime writes:          132 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      786590d7-a058-402d-b3a0-c3e0e6932320
Journal backup:           inode blocks
bamos commented 10 years ago

Top: Wolf's golden Bottom: Android

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem features:      has_journal ext_attr resize_inode filetype needs_recovery extent sparse_super large_file uninit_bg

rm
  dir_index
  flex_bg
  huge_file
  dir_nlink
  extra_isize
add
  needs_recovery

Filesystem features:      has_journal ext_attr resize_inode filetype needs_recovery extent sparse_super large_file uninit_bg
Filesystem features:      has_journal ext_attr resize_inode filetype needs_recovery extent sparse_super large_file uninit_bg
theonewolf commented 10 years ago

I created this to closely simulate the Android fs settings/numbers:

https://gist.github.com/theonewolf/0a700ac24d0c1d7e016f

Looks like things work fine with gammaray, perhaps this is a problem of needing to run fsck before scanning? gammaray can usually handle this case, but something odd with block group descriptors is happening here? I am not sure.

bamos commented 10 years ago

Edit: Skipping for now, likely not the cause.

theonewolf commented 10 years ago

Another thing I'd like to do is printout in my code debug stuff showing the ext4 superblock, and each BGD struct that is encountered. I don't have printout code for the superblock (it is actually in git history), but I wrote this simple bit today for BGDs (untested, but compiles):

https://gist.github.com/theonewolf/1e3c5fe2ae00543302a1

That might shed light onto why we can't get to a certain BGD. This will print out the BGD structure whenever any BGD is read (this is done while caching when we fail I think).

bamos commented 10 years ago
theonewolf commented 10 years ago

One small C program, one giant leap in quality.

So the nbd captured binary looks "corrupted" even to normal system tools like gdisk:

wolf@derecho ~> gdisk -l /home/bamos/data/gray/mmcblk0.raw
GPT fdisk (gdisk) version 0.8.10

Warning: File size is not a multiple of 512 bytes! Misbehavior is likely!
Warning: File size is not a multiple of 512 bytes! Misbehavior is likely!
Warning: File size is not a multiple of 512 bytes! Misbehavior is likely!
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: damaged

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
Disk /home/bamos/data/gray/mmcblk0.raw: 30798698 sectors, 14.7 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 52444E41-494F-2044-4D4D-43204449534B
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 30777310
Partitions will be aligned on 128-sector boundaries
Total free space is 735 sectors (367.5 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1             256             511   128.0 KiB   0700  xloader
   2            1024            8191   3.5 MiB     0700  sbl
   3            8192           49151   20.0 MiB    0700  efs
   4           49152           65535   8.0 MiB     0700  param
   5           65536           73727   4.0 MiB     0700  misc
   6           73728           81919   4.0 MiB     0700  dgs
   7           81920           98303   8.0 MiB     0700  boot
   8           98304          122751   11.9 MiB    0700  recovery
   9          122880          155647   16.0 MiB    0700  radio
  10          155648         1495039   654.0 MiB   0700  system
  11         1495040         2379775   432.0 MiB   0700  cache
  12         2379776        30777309   13.5 GiB    0700  userdata
  13          122752          122879   64.0 KiB    0700  metadata

But, look at what happens when I run the same tool on the netcp version:

wolf@derecho ~> gdisk -l /home/bamos/data/gray/mmcblk0-netcp.raw
GPT fdisk (gdisk) version 0.8.10

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /home/bamos/data/gray/mmcblk0-netcp.raw: 30777344 sectors, 14.7 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 52444E41-494F-2044-4D4D-43204449534B
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 30777310
Partitions will be aligned on 128-sector boundaries
Total free space is 735 sectors (367.5 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1             256             511   128.0 KiB   0700  xloader
   2            1024            8191   3.5 MiB     0700  sbl
   3            8192           49151   20.0 MiB    0700  efs
   4           49152           65535   8.0 MiB     0700  param
   5           65536           73727   4.0 MiB     0700  misc
   6           73728           81919   4.0 MiB     0700  dgs
   7           81920           98303   8.0 MiB     0700  boot
   8           98304          122751   11.9 MiB    0700  recovery
   9          122880          155647   16.0 MiB    0700  radio
  10          155648         1495039   654.0 MiB   0700  system
  11         1495040         2379775   432.0 MiB   0700  cache
  12         2379776        30777309   13.5 GiB    0700  userdata
  13          122752          122879   64.0 KiB    0700  metadata

Finally we have a good dump to work with :smile:! Glad I made netcp now, as it is paying off.

bamos commented 10 years ago

Awesome, great idea! :-)

bamos commented 10 years ago

By the way, the copy has finished if you want to further poke on this block device. I just ran the crawler on it and it unfortunately works well! Edit: Filesystem corruption doesn't make sense if it worked on the server. Maybe my block device is corrupted?

theonewolf commented 10 years ago

I think this points towards some difference between the emitted x86 code and the ARM binary.

I think the next step is to figure out exactly how the fseek got the numbers it believes to be correct, trace them (from block device, into gammaray), and figure out how they are being misinterpreted.

But yeah, gray-crawler appears to work perfectly on the dumped netcp bytes.

Also, Kiryong had some excellent insight: adb pull does everything we wanted sigh. I had to change ownership of /dev/block/mmcblk0 to shell:shell, but the pull appears to be working.

Oh well, netcp was still fun to code up (and only took a couple of hours).

theonewolf commented 10 years ago

Running off to a meeting, but @bamos can you run this on the phone:

https://gist.github.com/theonewolf/a457ee36d37e43f74bc5

bamos commented 10 years ago

Sure @theonewolf - unfortunately they look the same.

x86 output

/tmp/a457ee36d37e43f74bc5 [master*] » ./endian-test
p32[0] = 0x01
p32[1] = 0x02
p32[2] = 0x03
p32[3] = 0x04
p64[0] = 0x01
p64[1] = 0x02
p64[2] = 0x03
p64[3] = 0x04
p64[4] = 0x05
p64[5] = 0x06
p64[6] = 0x07
p64[7] = 0x08

ARM/Android Output

root@maguro:/data/gray # ./endian-test-arm                                     
p32[0] = 0x01
p32[1] = 0x02
p32[2] = 0x03
p32[3] = 0x04
p64[0] = 0x01
p64[1] = 0x02
p64[2] = 0x03
p64[3] = 0x04
p64[4] = 0x05
p64[5] = 0x06
p64[6] = 0x07
p64[7] = 0x08
bamos commented 10 years ago

Hi @theonewolf - I've added ext4 traces as a comment at https://gist.github.com/theonewolf/1e3c5fe2ae00543302a1

theonewolf commented 10 years ago

Glad we tracked this down to Android not respecting:

#define _FILE_OFFSET_BITS 64

So now we know that none of the fseeko operations were safe.

I have a new "safe" branch designed for this exotic Android environment:

https://github.com/cmusatyalab/gammaray/tree/safe-64bit-crawler

Note, particularly this is for Android: https://github.com/cmusatyalab/gammaray/commit/7a9e32a099ec2622445b0f4074223fcfc0dec35b

It's a sweeping change so I'm worried about regressions, but I'm hoping we'll find those as we go along.

bamos commented 10 years ago

Hi @theonewolf - awesome, thanks! The code management of this is getting a little messy. Are you ready to pull the GPT commits into the master. Check out the diffs here: https://github.com/wenluhu/gammaray-android/compare/cmusatyalab:master...gpt

For now, I'll merge your safe-64bit-crawler code into the master of this fork so we can resolve this and move on for this project.

bamos commented 10 years ago

Also, merging GPT code should also include the GUID fix from #18. I'll have this done shortly after the demo. :-)

theonewolf commented 10 years ago

I know, ideally we'd merge it into gammaray upstream now.

You think we should defer until #18? Also, maybe a final code cleanup (delete commented out code).