search

0x09 / hfsfuse

FUSE driver for HFS+ filesystems
Other
77 stars 13 forks source link

32-bit build fails to open large dmgs #9

Closed torarnv closed 4 years ago

torarnv commented 5 years ago

Hi there! I managed to do a 32-bit build by passing CFLAGS=-m32 LDFLAGS=-m32 to make (with some warnings), but the final binaries seem to fail to open a dmg that can be opened fine by the 64-bit build. 

root@5e9f2b8bc94f:/build# ./hfsdump /src/tests/data/100MB.dmg
Volume name: Empty
Journaled? 1
Readonly? 1
Offset: 0
volume header:
signature: H+
version: 4
attributes: hwlock 0 unmounted 0 badblocks 0 nocache 0 dirty 0 cnids recycled 0 journaled 1 swlock 0
last_mounting_version: HFSJ
journal_info_block: 2
date_created: Mon Sep 17 11:00:53 2018
date_modified: Mon Sep 17 09:00:55 2018
date_backedup: Thu Jan  1 00:00:00 1970
date_checked: Mon Sep 17 09:00:53 2018
file_count: 3
folder_count: 3
block_size: 4096
total_blocks: 24414
free_blocks: 23711
next_alloc_block: 4501
rsrc_clump_size: 65536
data_clump_size: 65536
next_cnid: 22
write_count: 3
encodings: 1
finderinfo:
    Boot directory ID: 0
    Startup parent directory ID: 0
    Display directory ID: 0
    OS classic system directory ID: 0
    OS X system directory ID: 0
    Volume unique ID: 6ffde41b5034bf29

root@5e9f2b8bc94f:/build# ./hfsdump /src/tests/data/8GB.dmg
could not open device
Couldn't open volume

Any ideas? has this been tested? 

cc -m32 -DFUSE_USE_VERSION=28 -D_FILE_OFFSET_BITS=64 -I /build/hfsfhuse/lib -c -o src/hfsfuse.o src/hfsfuse.c
src/hfsfuse.c: In function 'hfsfuse_open':
src/hfsfuse.c:68:13: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
  info->fh = (uint64_t)f;
             ^
src/hfsfuse.c: In function 'hfsfuse_release':
src/hfsfuse.c:74:22: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  struct hf_file* f = (struct hf_file*)info->fh;
                      ^
src/hfsfuse.c: In function 'hfsfuse_read':
src/hfsfuse.c:82:22: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  struct hf_file* f = (struct hf_file*)info->fh;
                      ^
src/hfsfuse.c: In function 'hfsfuse_fgetattr':
src/hfsfuse.c:112:22: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  struct hf_file* f = (struct hf_file*)info->fh;
                      ^
src/hfsfuse.c: In function 'hfsfuse_opendir':
src/hfsfuse.c:151:13: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
  info->fh = (uint64_t)d;
             ^
src/hfsfuse.c: In function 'hfsfuse_releasedir':
src/hfsfuse.c:156:21: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  struct hf_dir* d = (struct hf_dir*)info->fh;
                     ^
src/hfsfuse.c: In function 'hfsfuse_readdir':
src/hfsfuse.c:167:21: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  struct hf_dir* d = (struct hf_dir*)info->fh;
                     ^
src/hfsfuse.c: In function 'hfsfuse_readdir2':
src/hfsfuse.c:190:21: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  struct hf_dir* d = (struct hf_dir*)info->fh;
                     ^
0x09 commented 5 years ago

Thanks, this was originally developed for a 32 bit system so I'd expect/want it to work with -m32. The warnings are harmless -- info->fh is a uint64_t which is being used to store a 32 bit pointer in this case. They could be silenced by adding casts to/from uintptr_t. The error is because hfsfuse doesn't parse dmgs directly, the relevant partition needs to be extracted to an image first as described here.

torarnv commented 5 years ago

Thanks for your feedback!

The dmg was created with "no partition map", which should make it a raw HFS+ partition, just like an image, as far as I understand? As you can see, /hfsdump /src/tests/data/100MB.dmgworks fine, and the 8GB dmg works wine with a 64-bit build of hfsdump, without creating an image first.

torarnv commented 5 years ago

(I'm trying to use hfsfuse to test https://github.com/torarnv/sparsebundlefs, so converting the dmg to an image is not an option I'm afraid)

torarnv commented 5 years ago

E.g., 64-bit build on macOS:

❯ hdiutil create -size 100MB -type UDIF -fs HFS+ -layout NONE /tmp/foo.dmg
created: /tmp/foo.dmg

❯ ./hfsdump /tmp/foo.dmg
Volume name: untitled
Journaled? 0
Readonly? 1
Offset: 0
volume header:
signature: H+
version: 4
attributes: hwlock 0 unmounted 1 badblocks 0 nocache 0 dirty 0 cnids recycled 0 journaled 0 swlock 0
last_mounting_version: 10.0
journal_info_block: 0
date_created: Mon Sep 17 16:41:36 2018
date_modified: Mon Sep 17 14:41:36 2018
date_backedup: Thu Jan  1 01:00:00 1970
date_checked: Mon Sep 17 14:41:36 2018
file_count: 0
folder_count: 0
block_size: 4096
total_blocks: 25600
free_blocks: 24997
next_alloc_block: 4602
rsrc_clump_size: 65536
data_clump_size: 65536
next_cnid: 16
write_count: 0
encodings: 1
finderinfo:
    Boot directory ID: 0
    Startup parent directory ID: 0
    Display directory ID: 0
    OS classic system directory ID: 0
    OS X system directory ID: 0
    Volume unique ID: d07e927fad256fa4

❯ hdiutil create -size 8GB -type UDIF -fs HFS+ -layout NONE /tmp/8GB.dmg
created: /tmp/8GB.dmg

❯ ./hfsdump /tmp/8GB.dmg
Volume name: untitled
Journaled? 0
Readonly? 1
Offset: 0
volume header:
signature: H+
version: 4
attributes: hwlock 0 unmounted 1 badblocks 0 nocache 0 dirty 0 cnids recycled 0 journaled 0 swlock 0
last_mounting_version: 10.0
journal_info_block: 0
date_created: Mon Sep 17 16:42:23 2018
date_modified: Mon Sep 17 14:42:23 2018
date_backedup: Thu Jan  1 01:00:00 1970
date_checked: Mon Sep 17 14:42:23 2018
file_count: 0
folder_count: 0
block_size: 4096
total_blocks: 2097152
free_blocks: 2090174
next_alloc_block: 63297
rsrc_clump_size: 65536
data_clump_size: 65536
next_cnid: 16
write_count: 0
encodings: 1
finderinfo:
    Boot directory ID: 0
    Startup parent directory ID: 0
    Display directory ID: 0
    OS classic system directory ID: 0
    OS X system directory ID: 0
    Volume unique ID: 5d04133d5622f331
0x09 commented 5 years ago

Oh I see. That does seem like a bug, I'll look into it. Thanks!

torarnv commented 5 years ago

Thanks!! 🎉

0x09 commented 5 years ago

@torarnv I'm not able to reproduce this, is there anything that might be different about the image that's causing problems? Do you get the same result when building with WITH_UBLIO=none?

$ lipo -info hfsdump 
Non-fat file: hfsdump is architecture: i386

$ hdiutil create -size 8GB -type UDIF -fs HFS+ -layout NONE /tmp/8GB.dmg
.....................................................................................................................................................................................................................................
created: /tmp/8GB.dmg

$ ./hfsdump /tmp/8GB.dmg 
Volume name: untitled
Journaled? 0
Readonly? 1
Offset: 0
volume header:
signature: H+
version: 4
attributes: hwlock 0 unmounted 1 badblocks 0 nocache 0 dirty 0 cnids recycled 0 journaled 0 swlock 0
last_mounting_version: 10.0
journal_info_block: 0
date_created: Tue Sep 18 17:32:09 2018
date_modified: Tue Sep 18 21:32:09 2018
date_backedup: Wed Dec 31 19:00:00 1969
date_checked: Tue Sep 18 21:32:09 2018
file_count: 0
folder_count: 0
block_size: 4096
total_blocks: 2097152
free_blocks: 2090174
next_alloc_block: 63297
rsrc_clump_size: 65536
data_clump_size: 65536
next_cnid: 16
write_count: 0
encodings: 1
finderinfo:
    Boot directory ID: 0
    Startup parent directory ID: 0
    Display directory ID: 0
    OS classic system directory ID: 0
    OS X system directory ID: 0
    Volume unique ID: cd9ce94c71b0440f

$ ./hfsdump /tmp/8GB.dmg read /
.DS_Store
.fseventsd
torarnv commented 5 years ago

Hey hey! Thanks for testing!! You are right, I can't reproduce it with a 32-bit build on macOS (CFLAGS=-m32 LDFLAGS=-m32 make) 🤔

But using the following Dockerfile, I can reproduce:

FROM ubuntu:16.04

ARG arch=i386

RUN dpkg --add-architecture $arch && \
    apt-get update && \
    apt-get install -y \
        build-essential \
        git \
        file \
        g++-multilib \
        pkg-config:$arch \
        libfuse-dev:$arch \
        fuse:$arch

Giving:

❯ docker run -it --rm -v $(pwd):/hfsfuse 3a373be4a49d
root@1f96f4ec4f17:/# cd /hfsfuse/
root@1f96f4ec4f17:/hfsfuse# ls
COPYING  Dockerfile  Makefile  README.md  lib  src
root@1f96f4ec4f17:/hfsfuse# CFLAGS=-m32 LDFLAGS=-m32 make
echo \#define HFSFUSE_VERSION_STRING  > src/version.h
cc -O3 -std=gnu11 -m32 -DFUSE_USE_VERSION=28 -D_FILE_OFFSET_BITS=64 -I /hfsfuse/lib -c -o src/hfsfuse.o src/hfsfuse.c
src/hfsfuse.c: In function 'hfsfuse_open':
src/hfsfuse.c:68:13: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
  info->fh = (uint64_t)f;
             ^
src/hfsfuse.c: In function 'hfsfuse_release':
src/hfsfuse.c:74:22: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  struct hf_file* f = (struct hf_file*)info->fh;
                      ^
src/hfsfuse.c: In function 'hfsfuse_read':
src/hfsfuse.c:82:22: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  struct hf_file* f = (struct hf_file*)info->fh;
                      ^
src/hfsfuse.c: In function 'hfsfuse_fgetattr':
src/hfsfuse.c:112:22: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  struct hf_file* f = (struct hf_file*)info->fh;
                      ^
src/hfsfuse.c: In function 'hfsfuse_opendir':
src/hfsfuse.c:151:13: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
  info->fh = (uint64_t)d;
             ^
src/hfsfuse.c: In function 'hfsfuse_releasedir':
src/hfsfuse.c:156:21: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  struct hf_dir* d = (struct hf_dir*)info->fh;
                     ^
src/hfsfuse.c: In function 'hfsfuse_readdir':
src/hfsfuse.c:167:21: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  struct hf_dir* d = (struct hf_dir*)info->fh;
                     ^
src/hfsfuse.c: In function 'hfsfuse_readdir2':
src/hfsfuse.c:190:21: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  struct hf_dir* d = (struct hf_dir*)info->fh;
                     ^
make -C lib/libhfsuser/
make[1]: Entering directory '/hfsfuse/lib/libhfsuser'
cc -O3 -std=gnu11 -m32 -DHAVE_UBLIO -DHAVE_UTF8PROC -I /hfsfuse/lib -c -o hfsuser.o hfsuser.c
cc -O3 -std=gnu11 -m32 -DHAVE_UBLIO -DHAVE_UTF8PROC -I /hfsfuse/lib -c -o features.o features.c
cc -O3 -std=gnu11 -m32 -DHAVE_UBLIO -DHAVE_UTF8PROC -I /hfsfuse/lib -c -o cache.o cache.c
ar rcs libhfsuser.a hfsuser.o features.o cache.o
ranlib libhfsuser.a
make[1]: Leaving directory '/hfsfuse/lib/libhfsuser'
make -C lib/libhfs/
make[1]: Entering directory '/hfsfuse/lib/libhfs'
cc -O3 -std=gnu11 -m32 -c -o libhfs.o libhfs.c
cc -O3 -std=gnu11 -m32 -c -o unicode.o unicode.c
cc -O3 -std=gnu11 -m32 -c -o endian.o endian.c
ar rcs libhfs.a libhfs.o unicode.o endian.o
ranlib libhfs.a
make[1]: Leaving directory '/hfsfuse/lib/libhfs'
make -C lib/ublio/
make[1]: Entering directory '/hfsfuse/lib/ublio'
cc -g -O2 -Wall -O3 -std=gnu11 -m32 -c ublio.c -o ublio.o
ar rcs libublio.a ublio.o
make[1]: Leaving directory '/hfsfuse/lib/ublio'
make -C lib/utf8proc/
make[1]: Entering directory '/hfsfuse/lib/utf8proc'
cc -O3 -std=gnu11 -m32 -fPIC -std=c99 -Wall -Wmissing-prototypes -pedantic -DUTF8PROC_EXPORTS -c -o utf8proc.o utf8proc.c
rm -f libutf8proc.a
ar rs libutf8proc.a utf8proc.o
ar: creating libutf8proc.a
cc -m32 -shared -o libutf8proc.so.2.0.0 -Wl,-soname -Wl,libutf8proc.so.2 utf8proc.o
chmod a-x libutf8proc.so.2.0.0
ln -f -s libutf8proc.so.2.0.0 libutf8proc.so
ln -f -s libutf8proc.so.2.0.0 libutf8proc.so.2
make[1]: Leaving directory '/hfsfuse/lib/utf8proc'
cc -O3 -std=gnu11 -m32  -o hfsfuse src/hfsfuse.o lib/libhfsuser/libhfsuser.a lib/libhfs/libhfs.a lib/ublio/libublio.a lib/utf8proc/libutf8proc.a -lfuse -lpthread
cc -O3 -std=gnu11 -m32 -DFUSE_USE_VERSION=28 -D_FILE_OFFSET_BITS=64 -I /hfsfuse/lib -c -o src/hfsdump.o src/hfsdump.c
cc -O3 -std=gnu11 -m32  -o hfsdump src/hfsdump.o lib/libhfsuser/libhfsuser.a lib/libhfs/libhfs.a lib/ublio/libublio.a lib/utf8proc/libutf8proc.a -lpthread
root@1f96f4ec4f17:/hfsfuse# ls
COPYING  Dockerfile  Makefile  README.md  hfsdump  hfsfuse  lib  src
root@1f96f4ec4f17:/hfsfuse# file hfsdump
hfsdump: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=1e2b3752bb5db0d8dcffb0339713bb3c33c3b867, not stripped
root@1f96f4ec4f17:/hfsfuse# ls
100MB.dmg  8GB.dmg  COPYING  Dockerfile  Makefile  README.md  hfsdump  hfsfuse  lib  src
root@1f96f4ec4f17:/hfsfuse# ./hfsdump 100MB.dmg
Volume name: untitled
Journaled? 0
Readonly? 1
Offset: 0
volume header:
signature: H+
version: 4
attributes: hwlock 0 unmounted 1 badblocks 0 nocache 0 dirty 0 cnids recycled 0 journaled 0 swlock 0
last_mounting_version: 10.0
journal_info_block: 0
date_created: Mon Sep 17 14:41:36 2018
date_modified: Mon Sep 17 12:41:36 2018
date_backedup: Thu Jan  1 00:00:00 1970
date_checked: Mon Sep 17 12:41:36 2018
file_count: 0
folder_count: 0
block_size: 4096
total_blocks: 25600
free_blocks: 24997
next_alloc_block: 4602
rsrc_clump_size: 65536
data_clump_size: 65536
next_cnid: 16
write_count: 0
encodings: 1
finderinfo:
    Boot directory ID: 0
    Startup parent directory ID: 0
    Display directory ID: 0
    OS classic system directory ID: 0
    OS X system directory ID: 0
    Volume unique ID: d07e927fad256fa4
root@1f96f4ec4f17:/hfsfuse# ./hfsdump 8GB.dmg
could not open device
Couldn't open volume
root@1f96f4ec4f17:/hfsfuse#

Same thing with WITH_UBLIO=none I'm afraid.

Can you reproduce using the Dockerfile? Is there anything I can do to debug it further on my end? Thanks!

torarnv commented 5 years ago

Here's the strace:

root@91e0fe24e4d9:/hfsfuse# strace ./hfsdump 8GB.dmg
execve("./hfsdump", ["./hfsdump", "8GB.dmg"], [/* 11 vars */]) = 0
strace: [ Process PID=47 runs in 32 bit mode. ]
brk(NULL)                               = 0x9780000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xf773c000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=29492, ...}) = 0
mmap2(NULL, 29492, PROT_READ, MAP_PRIVATE, 3, 0) = 0xf7734000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260O\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=134864, ...}) = 0
mmap2(NULL, 115296, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xf7717000
mmap2(0xf7730000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0xf7730000
mmap2(0xf7732000, 4704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xf7732000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\3\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320\207\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1786484, ...}) = 0
mmap2(NULL, 1792540, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xf7561000
mmap2(0xf7711000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1af000) = 0xf7711000
mmap2(0xf7714000, 10780, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xf7714000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xf7560000
set_thread_area({entry_number:-1, base_addr:0xf7560700, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0 (entry_number:12)
mprotect(0xf7711000, 8192, PROT_READ)   = 0
mprotect(0xf7730000, 4096, PROT_READ)   = 0
mprotect(0x80cc000, 4096, PROT_READ)    = 0
mprotect(0xf7763000, 4096, PROT_READ)   = 0
munmap(0xf7734000, 29492)               = 0
set_tid_address(0xf7560768)             = 47
set_robust_list(0xf7560770, 12)         = 0
rt_sigaction(SIGRTMIN, {0xf771b9c0, [], SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0xf771ba40, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
ugetrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
uname({sysname="Linux", nodename="91e0fe24e4d9", ...}) = 0
brk(NULL)                               = 0x9780000
brk(0x97a1000)                          = 0x97a1000
open("8GB.dmg", O_RDONLY)               = -1 EOVERFLOW (Value too large for defined data type)
write(2, "could not open device", 21could not open device)   = 21
write(2, "\n", 1
)                       = 1
write(2, "Couldn't open volume\n", 21Couldn't open volume
)  = 21
exit_group(1)                           = ?
+++ exited with 1 +++
torarnv commented 5 years ago

Hurray, making sure we include large-file-support, with CFLAGS="-m32 -D_FILE_OFFSET_BITS=64" LDFLAGS=-m32 make, solved the issue!!

torarnv commented 5 years ago

I see you're setting it unconditionally for the main binaries, but not for the libs. Perhaps something like this:

diff --git i/Makefile w/Makefile
index b100559..66e8d2b 100644
--- i/Makefile
+++ w/Makefile
@@ -61,7 +61,7 @@ all: hfsfuse hfsdump
    $(CC) $(CFLAGS) $(FUSE_FLAGS) $(INCLUDE) -c -o $*.o $^

 $(LIBS): always_check
-   $(MAKE) -C $(dir $@)
+   $(MAKE) -C $(dir $@) CFLAGS="-D_FILE_OFFSET_BITS=64"

 lib: $(LIBS)
torarnv commented 5 years ago

Hmm, no, that's not going to work if passing CFLAGS=-m32 LDFLAGS=-m32 make.

0x09 commented 5 years ago

@torarnv I see what's going on here. The glibc headers actually redefine the open call based on whether this is set to one with large file support, which indeed affects libhfsuser. Since FILE_OFFSET_BITS is unused on other platforms I think it's fine to enable this unconditionally there.

Edit: can you pull and try building with the updated Makefile? Should be resolved with https://github.com/0x09/hfsfuse/commit/facc800cdd53a4bd4cc711b2b87a39dacd5688fb