Closed torarnv closed 4 years ago
Thanks, this was originally developed for a 32 bit system so I'd expect/want it to work with -m32. The warnings are harmless -- info->fh is a uint64_t which is being used to store a 32 bit pointer in this case. They could be silenced by adding casts to/from uintptr_t. The error is because hfsfuse doesn't parse dmgs directly, the relevant partition needs to be extracted to an image first as described here.
Thanks for your feedback!
The dmg was created with "no partition map", which should make it a raw HFS+ partition, just like an image, as far as I understand? As you can see, /hfsdump /src/tests/data/100MB.dmg
works fine, and the 8GB dmg works wine with a 64-bit build of hfsdump, without creating an image first.
(I'm trying to use hfsfuse to test https://github.com/torarnv/sparsebundlefs, so converting the dmg to an image is not an option I'm afraid)
E.g., 64-bit build on macOS:
❯ hdiutil create -size 100MB -type UDIF -fs HFS+ -layout NONE /tmp/foo.dmg
created: /tmp/foo.dmg
❯ ./hfsdump /tmp/foo.dmg
Volume name: untitled
Journaled? 0
Readonly? 1
Offset: 0
volume header:
signature: H+
version: 4
attributes: hwlock 0 unmounted 1 badblocks 0 nocache 0 dirty 0 cnids recycled 0 journaled 0 swlock 0
last_mounting_version: 10.0
journal_info_block: 0
date_created: Mon Sep 17 16:41:36 2018
date_modified: Mon Sep 17 14:41:36 2018
date_backedup: Thu Jan 1 01:00:00 1970
date_checked: Mon Sep 17 14:41:36 2018
file_count: 0
folder_count: 0
block_size: 4096
total_blocks: 25600
free_blocks: 24997
next_alloc_block: 4602
rsrc_clump_size: 65536
data_clump_size: 65536
next_cnid: 16
write_count: 0
encodings: 1
finderinfo:
Boot directory ID: 0
Startup parent directory ID: 0
Display directory ID: 0
OS classic system directory ID: 0
OS X system directory ID: 0
Volume unique ID: d07e927fad256fa4
❯ hdiutil create -size 8GB -type UDIF -fs HFS+ -layout NONE /tmp/8GB.dmg
created: /tmp/8GB.dmg
❯ ./hfsdump /tmp/8GB.dmg
Volume name: untitled
Journaled? 0
Readonly? 1
Offset: 0
volume header:
signature: H+
version: 4
attributes: hwlock 0 unmounted 1 badblocks 0 nocache 0 dirty 0 cnids recycled 0 journaled 0 swlock 0
last_mounting_version: 10.0
journal_info_block: 0
date_created: Mon Sep 17 16:42:23 2018
date_modified: Mon Sep 17 14:42:23 2018
date_backedup: Thu Jan 1 01:00:00 1970
date_checked: Mon Sep 17 14:42:23 2018
file_count: 0
folder_count: 0
block_size: 4096
total_blocks: 2097152
free_blocks: 2090174
next_alloc_block: 63297
rsrc_clump_size: 65536
data_clump_size: 65536
next_cnid: 16
write_count: 0
encodings: 1
finderinfo:
Boot directory ID: 0
Startup parent directory ID: 0
Display directory ID: 0
OS classic system directory ID: 0
OS X system directory ID: 0
Volume unique ID: 5d04133d5622f331
Oh I see. That does seem like a bug, I'll look into it. Thanks!
Thanks!! 🎉
@torarnv I'm not able to reproduce this, is there anything that might be different about the image that's causing problems? Do you get the same result when building with WITH_UBLIO=none?
$ lipo -info hfsdump
Non-fat file: hfsdump is architecture: i386
$ hdiutil create -size 8GB -type UDIF -fs HFS+ -layout NONE /tmp/8GB.dmg
.....................................................................................................................................................................................................................................
created: /tmp/8GB.dmg
$ ./hfsdump /tmp/8GB.dmg
Volume name: untitled
Journaled? 0
Readonly? 1
Offset: 0
volume header:
signature: H+
version: 4
attributes: hwlock 0 unmounted 1 badblocks 0 nocache 0 dirty 0 cnids recycled 0 journaled 0 swlock 0
last_mounting_version: 10.0
journal_info_block: 0
date_created: Tue Sep 18 17:32:09 2018
date_modified: Tue Sep 18 21:32:09 2018
date_backedup: Wed Dec 31 19:00:00 1969
date_checked: Tue Sep 18 21:32:09 2018
file_count: 0
folder_count: 0
block_size: 4096
total_blocks: 2097152
free_blocks: 2090174
next_alloc_block: 63297
rsrc_clump_size: 65536
data_clump_size: 65536
next_cnid: 16
write_count: 0
encodings: 1
finderinfo:
Boot directory ID: 0
Startup parent directory ID: 0
Display directory ID: 0
OS classic system directory ID: 0
OS X system directory ID: 0
Volume unique ID: cd9ce94c71b0440f
$ ./hfsdump /tmp/8GB.dmg read /
.DS_Store
.fseventsd
Hey hey! Thanks for testing!! You are right, I can't reproduce it with a 32-bit build on macOS (CFLAGS=-m32 LDFLAGS=-m32 make
) 🤔
But using the following Dockerfile, I can reproduce:
FROM ubuntu:16.04
ARG arch=i386
RUN dpkg --add-architecture $arch && \
apt-get update && \
apt-get install -y \
build-essential \
git \
file \
g++-multilib \
pkg-config:$arch \
libfuse-dev:$arch \
fuse:$arch
Giving:
❯ docker run -it --rm -v $(pwd):/hfsfuse 3a373be4a49d
root@1f96f4ec4f17:/# cd /hfsfuse/
root@1f96f4ec4f17:/hfsfuse# ls
COPYING Dockerfile Makefile README.md lib src
root@1f96f4ec4f17:/hfsfuse# CFLAGS=-m32 LDFLAGS=-m32 make
echo \#define HFSFUSE_VERSION_STRING > src/version.h
cc -O3 -std=gnu11 -m32 -DFUSE_USE_VERSION=28 -D_FILE_OFFSET_BITS=64 -I /hfsfuse/lib -c -o src/hfsfuse.o src/hfsfuse.c
src/hfsfuse.c: In function 'hfsfuse_open':
src/hfsfuse.c:68:13: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
info->fh = (uint64_t)f;
^
src/hfsfuse.c: In function 'hfsfuse_release':
src/hfsfuse.c:74:22: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
struct hf_file* f = (struct hf_file*)info->fh;
^
src/hfsfuse.c: In function 'hfsfuse_read':
src/hfsfuse.c:82:22: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
struct hf_file* f = (struct hf_file*)info->fh;
^
src/hfsfuse.c: In function 'hfsfuse_fgetattr':
src/hfsfuse.c:112:22: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
struct hf_file* f = (struct hf_file*)info->fh;
^
src/hfsfuse.c: In function 'hfsfuse_opendir':
src/hfsfuse.c:151:13: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
info->fh = (uint64_t)d;
^
src/hfsfuse.c: In function 'hfsfuse_releasedir':
src/hfsfuse.c:156:21: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
struct hf_dir* d = (struct hf_dir*)info->fh;
^
src/hfsfuse.c: In function 'hfsfuse_readdir':
src/hfsfuse.c:167:21: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
struct hf_dir* d = (struct hf_dir*)info->fh;
^
src/hfsfuse.c: In function 'hfsfuse_readdir2':
src/hfsfuse.c:190:21: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
struct hf_dir* d = (struct hf_dir*)info->fh;
^
make -C lib/libhfsuser/
make[1]: Entering directory '/hfsfuse/lib/libhfsuser'
cc -O3 -std=gnu11 -m32 -DHAVE_UBLIO -DHAVE_UTF8PROC -I /hfsfuse/lib -c -o hfsuser.o hfsuser.c
cc -O3 -std=gnu11 -m32 -DHAVE_UBLIO -DHAVE_UTF8PROC -I /hfsfuse/lib -c -o features.o features.c
cc -O3 -std=gnu11 -m32 -DHAVE_UBLIO -DHAVE_UTF8PROC -I /hfsfuse/lib -c -o cache.o cache.c
ar rcs libhfsuser.a hfsuser.o features.o cache.o
ranlib libhfsuser.a
make[1]: Leaving directory '/hfsfuse/lib/libhfsuser'
make -C lib/libhfs/
make[1]: Entering directory '/hfsfuse/lib/libhfs'
cc -O3 -std=gnu11 -m32 -c -o libhfs.o libhfs.c
cc -O3 -std=gnu11 -m32 -c -o unicode.o unicode.c
cc -O3 -std=gnu11 -m32 -c -o endian.o endian.c
ar rcs libhfs.a libhfs.o unicode.o endian.o
ranlib libhfs.a
make[1]: Leaving directory '/hfsfuse/lib/libhfs'
make -C lib/ublio/
make[1]: Entering directory '/hfsfuse/lib/ublio'
cc -g -O2 -Wall -O3 -std=gnu11 -m32 -c ublio.c -o ublio.o
ar rcs libublio.a ublio.o
make[1]: Leaving directory '/hfsfuse/lib/ublio'
make -C lib/utf8proc/
make[1]: Entering directory '/hfsfuse/lib/utf8proc'
cc -O3 -std=gnu11 -m32 -fPIC -std=c99 -Wall -Wmissing-prototypes -pedantic -DUTF8PROC_EXPORTS -c -o utf8proc.o utf8proc.c
rm -f libutf8proc.a
ar rs libutf8proc.a utf8proc.o
ar: creating libutf8proc.a
cc -m32 -shared -o libutf8proc.so.2.0.0 -Wl,-soname -Wl,libutf8proc.so.2 utf8proc.o
chmod a-x libutf8proc.so.2.0.0
ln -f -s libutf8proc.so.2.0.0 libutf8proc.so
ln -f -s libutf8proc.so.2.0.0 libutf8proc.so.2
make[1]: Leaving directory '/hfsfuse/lib/utf8proc'
cc -O3 -std=gnu11 -m32 -o hfsfuse src/hfsfuse.o lib/libhfsuser/libhfsuser.a lib/libhfs/libhfs.a lib/ublio/libublio.a lib/utf8proc/libutf8proc.a -lfuse -lpthread
cc -O3 -std=gnu11 -m32 -DFUSE_USE_VERSION=28 -D_FILE_OFFSET_BITS=64 -I /hfsfuse/lib -c -o src/hfsdump.o src/hfsdump.c
cc -O3 -std=gnu11 -m32 -o hfsdump src/hfsdump.o lib/libhfsuser/libhfsuser.a lib/libhfs/libhfs.a lib/ublio/libublio.a lib/utf8proc/libutf8proc.a -lpthread
root@1f96f4ec4f17:/hfsfuse# ls
COPYING Dockerfile Makefile README.md hfsdump hfsfuse lib src
root@1f96f4ec4f17:/hfsfuse# file hfsdump
hfsdump: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=1e2b3752bb5db0d8dcffb0339713bb3c33c3b867, not stripped
root@1f96f4ec4f17:/hfsfuse# ls
100MB.dmg 8GB.dmg COPYING Dockerfile Makefile README.md hfsdump hfsfuse lib src
root@1f96f4ec4f17:/hfsfuse# ./hfsdump 100MB.dmg
Volume name: untitled
Journaled? 0
Readonly? 1
Offset: 0
volume header:
signature: H+
version: 4
attributes: hwlock 0 unmounted 1 badblocks 0 nocache 0 dirty 0 cnids recycled 0 journaled 0 swlock 0
last_mounting_version: 10.0
journal_info_block: 0
date_created: Mon Sep 17 14:41:36 2018
date_modified: Mon Sep 17 12:41:36 2018
date_backedup: Thu Jan 1 00:00:00 1970
date_checked: Mon Sep 17 12:41:36 2018
file_count: 0
folder_count: 0
block_size: 4096
total_blocks: 25600
free_blocks: 24997
next_alloc_block: 4602
rsrc_clump_size: 65536
data_clump_size: 65536
next_cnid: 16
write_count: 0
encodings: 1
finderinfo:
Boot directory ID: 0
Startup parent directory ID: 0
Display directory ID: 0
OS classic system directory ID: 0
OS X system directory ID: 0
Volume unique ID: d07e927fad256fa4
root@1f96f4ec4f17:/hfsfuse# ./hfsdump 8GB.dmg
could not open device
Couldn't open volume
root@1f96f4ec4f17:/hfsfuse#
Same thing with WITH_UBLIO=none
I'm afraid.
Can you reproduce using the Dockerfile? Is there anything I can do to debug it further on my end? Thanks!
Here's the strace:
root@91e0fe24e4d9:/hfsfuse# strace ./hfsdump 8GB.dmg
execve("./hfsdump", ["./hfsdump", "8GB.dmg"], [/* 11 vars */]) = 0
strace: [ Process PID=47 runs in 32 bit mode. ]
brk(NULL) = 0x9780000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xf773c000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=29492, ...}) = 0
mmap2(NULL, 29492, PROT_READ, MAP_PRIVATE, 3, 0) = 0xf7734000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260O\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=134864, ...}) = 0
mmap2(NULL, 115296, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xf7717000
mmap2(0xf7730000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0xf7730000
mmap2(0xf7732000, 4704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xf7732000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\3\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320\207\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1786484, ...}) = 0
mmap2(NULL, 1792540, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xf7561000
mmap2(0xf7711000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1af000) = 0xf7711000
mmap2(0xf7714000, 10780, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xf7714000
close(3) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xf7560000
set_thread_area({entry_number:-1, base_addr:0xf7560700, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0 (entry_number:12)
mprotect(0xf7711000, 8192, PROT_READ) = 0
mprotect(0xf7730000, 4096, PROT_READ) = 0
mprotect(0x80cc000, 4096, PROT_READ) = 0
mprotect(0xf7763000, 4096, PROT_READ) = 0
munmap(0xf7734000, 29492) = 0
set_tid_address(0xf7560768) = 47
set_robust_list(0xf7560770, 12) = 0
rt_sigaction(SIGRTMIN, {0xf771b9c0, [], SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0xf771ba40, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
ugetrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
uname({sysname="Linux", nodename="91e0fe24e4d9", ...}) = 0
brk(NULL) = 0x9780000
brk(0x97a1000) = 0x97a1000
open("8GB.dmg", O_RDONLY) = -1 EOVERFLOW (Value too large for defined data type)
write(2, "could not open device", 21could not open device) = 21
write(2, "\n", 1
) = 1
write(2, "Couldn't open volume\n", 21Couldn't open volume
) = 21
exit_group(1) = ?
+++ exited with 1 +++
Hurray, making sure we include large-file-support, with CFLAGS="-m32 -D_FILE_OFFSET_BITS=64" LDFLAGS=-m32 make
, solved the issue!!
I see you're setting it unconditionally for the main binaries, but not for the libs. Perhaps something like this:
diff --git i/Makefile w/Makefile
index b100559..66e8d2b 100644
--- i/Makefile
+++ w/Makefile
@@ -61,7 +61,7 @@ all: hfsfuse hfsdump
$(CC) $(CFLAGS) $(FUSE_FLAGS) $(INCLUDE) -c -o $*.o $^
$(LIBS): always_check
- $(MAKE) -C $(dir $@)
+ $(MAKE) -C $(dir $@) CFLAGS="-D_FILE_OFFSET_BITS=64"
lib: $(LIBS)
Hmm, no, that's not going to work if passing CFLAGS=-m32 LDFLAGS=-m32 make
.
@torarnv I see what's going on here. The glibc headers actually redefine the open
call based on whether this is set to one with large file support, which indeed affects libhfsuser. Since FILE_OFFSET_BITS is unused on other platforms I think it's fine to enable this unconditionally there.
Edit: can you pull and try building with the updated Makefile? Should be resolved with https://github.com/0x09/hfsfuse/commit/facc800cdd53a4bd4cc711b2b87a39dacd5688fb
Hi there! I managed to do a 32-bit build by passing
CFLAGS=-m32 LDFLAGS=-m32
to make (with some warnings), but the final binaries seem to fail to open a dmg that can be opened fine by the 64-bit build.Any ideas? has this been tested?