Open GoogleCodeExporter opened 9 years ago
I'm a little worried by this. I haven't seen it on any other platform. Can
you tell
me the compiler version? Issue 51 is related to a compiler bug as well. I am
not
familiar with the arm/xscale platform. What is xscale?
Original comment by josh.mac...@gmail.com
on 29 Nov 2007 at 4:55
XScale is an ARM processor (http://en.wikipedia.org/wiki/Xscale).
The compiler I used:
Configured with: /work/crosstool-0.27/build/arm-linux/gcc-3.4.1-glibc-2.3.2/gcc-
3.4.1/configure --target=arm-linux --host=i686-host_pc-linux-gnu
--prefix=/usr/local/
arm/3.4.1 --with-headers=/usr/local/arm/3.4.1/arm-linux/include
--with-local-prefix=/
usr/local/arm/3.4.1/arm-linux --disable-nls --enable-threads=posix --enable-
symvers=gnu --enable-__cxa_atexit --enable-languages=c,c++ --enable-shared
--enable-
c99 --enable-long-long
Thread model: posix
gcc version 3.4.1
What the official debian binaries were compiled with I don't know. But at least
the
error is consistent with everything I've tried so far. I even tried with the
compiler option I use for all my other ARM programs:
-Wall -fomit-frame-pointer -finline-functions -funroll-loops
-falign-functions=2 -
falign-loops=2 -falign-jumps=2 -mcpu=xscale -mtune=xscale
no difference...
Compared to the expected output everything seems to be shifted by two bytes. Is
there anything I can do to provide you with further clues for this?
Original comment by thez...@gmail.com
on 29 Nov 2007 at 5:38
Can you try compiling without any optimization, and re-run the test?
Also, -fomit-frame-pointer has caused problems in issue 51.
I do a lot of development on gcc 3.4, so this seems like a real issue.
Original comment by josh.mac...@gmail.com
on 30 Nov 2007 at 4:16
xdelta3 compiled with no optimizations and xdelta3-debug produce the same error
as
well.
Original comment by thez...@gmail.com
on 30 Nov 2007 at 8:29
If you could, I'm interested in the output of "xdelta3 config". I'm not sure
what
I'm looking for, but I'm interested.
Original comment by josh.mac...@gmail.com
on 1 Dec 2007 at 5:05
I believe this and issue 51 are caused by the same underlying issue,
uninitialized
data. I will keep you posted.
Original comment by josh.mac...@gmail.com
on 2 Dec 2007 at 7:15
Ok, here's the "xdelta3 config" output for version 3.0s:
Xdelta version 3.0s, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
Original comment by thez...@gmail.com
on 3 Dec 2007 at 8:15
Would you mind testing with SVN 207? I believe the problem is solved, there was
a bug
introduced in 3.0s.
Original comment by josh.mac...@gmail.com
on 5 Dec 2007 at 3:55
The test also fails with versions k and q as stated above.
Same thing unfortunately:
root@familiar:/tmp# ./xdelta3 test
Xdelta version 3.0s, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t.....................................
success
xdelta3: testing
encode_decode_uint64_t..........................................................
.........
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10
C14/
6@12 failed: wrong result: XD3_INTERNAL
Original comment by thez...@gmail.com
on 5 Dec 2007 at 8:08
Can you try a few more tests? The xdelta3-32 binary compiles with a 32-bit file
offset, and its test might shed some light on this. I've released 3.0t now,
and this
is my highest priority bug report.
Original comment by josh.mac...@gmail.com
on 7 Dec 2007 at 5:55
Ok, I can do this on monday at the earliest.
Original comment by thez...@gmail.com
on 8 Dec 2007 at 1:24
Run with svn rev 210
root@familiar:/tmp# ./xdelta3-32 test
Xdelta version 3.0t, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=1
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=0
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=4
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t.....................................
success
xdelta3: testing
encode_decode_uint64_t..........................................................
.........
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10
C14/
6@12 failed: wrong result: XD3_INTERNAL
root@familiar:/tmp# ./xdelta3 test
Xdelta version 3.0t, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t.....................................
success
xdelta3: testing
encode_decode_uint64_t..........................................................
.........
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10
C14/
6@12 failed: wrong result: XD3_INTERNAL
Original comment by thez...@gmail.com
on 10 Dec 2007 at 9:49
Thanks. I will go study the code and see if I can guess at a cause.
Original comment by josh.mac...@gmail.com
on 10 Dec 2007 at 6:11
I have a theory. ARM processors do not support unaligned access the way x86
does.
There is an unaligned access which may have contributed to this problem. Could
you
please run "xdelta3-debug test" and see if it produces the same errors? I am
hoping
it will crash with an assertion prior to the test failure.
SVN 211 has a few cleanups that should not make a difference, but it would be
best if
you could test with the latest version.
I will go about cleaning up the unaligned accesses using an memcpy for
platforms that
do not support efficient unaligned access.
Original comment by josh.mac...@gmail.com
on 13 Dec 2007 at 6:31
It produces the same output as the other tests. But then again it doesn't seem
to
violate any bounds. The string match result is offset by two bytes but seems ok
otherwise.
root@familiar:/tmp# ./xdelta3-debug test
Xdelta version 3.0t, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=1
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=1
XD3_ENCODER=1
XD3_POSIX=0
XD3_STDIO=1
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t.....................................
success
xdelta3: testing
encode_decode_uint64_t..........................................................
.........
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing address_cache (XD3_ALT_CODE_TABLE)... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10
C14/
6@12 failed: wrong result: XD3_INTERNAL
Original comment by thez...@gmail.com
on 13 Dec 2007 at 8:46
It is finding a correct match, but only when it starts at 12, not at 10. I
believe
this is because 12 is word-aligned and 10 is not. Let me see if I can add some
debugging (to diagnose further) and/or correct the problem (the solution
involves
memcpy).
Original comment by josh.mac...@gmail.com
on 14 Dec 2007 at 5:40
SVN 212 may solve the problem.
To be EXTRA sure, you could add -DUNALIGNED_OK=0 to CFLAGS,
but I've set things so that only __i386__ and similar defs
will set that variable by default.
Original comment by josh.mac...@gmail.com
on 14 Dec 2007 at 6:37
Good job, that fixed it :)
Another failing test:
xdelta3: testing no_output...command was ./xdelta3 -q -f -e
/tmp/xdtest.target.3 /
tmp/xdtest.noperm.3 failed: command should have not succeeded: XD3_INTERNAL
Do you think it's related to that one or should I open a new ticket?
Original comment by thez...@gmail.com
on 14 Dec 2007 at 1:04
We can keep this ticket open.
The test that fails now is related to file permissions. Are you running as
root,
possibly? Are there file permissions?
Original comment by josh.mac...@gmail.com
on 14 Dec 2007 at 6:19
Original comment by josh.mac...@gmail.com
on 14 Dec 2007 at 6:40
Actually I want to keep this issue open for another reason.
Now that we know the string_matching problem is fixed, I'd like to test a more
efficient solution. Stay tuned.
Original comment by josh.mac...@gmail.com
on 14 Dec 2007 at 7:30
Argh--I just cleaned up the UNALIGNED_OK=0 case while sitting at the laundromat
and
spill bleach on my computer (bad) on the way home!
Original comment by josh.mac...@gmail.com
on 14 Dec 2007 at 10:48
Aw, shit. Hope it still works...
root is actually the only user on the device... I created a test user now and
with
that it works alright. Next fail:
xdelta3: testing recode_command...xdelta3: printhdr command: /tmp/xdelta3
printhdr -
f /tmp/xdtest.copy.17 /tmp/xdtest.recon2.17 failed: printhdr cmd failed:
XD3_INTERNAL
Original comment by thez...@gmail.com
on 15 Dec 2007 at 12:04
There's a sizzling sound from the bleach on the battery contacts. Not good.
Oh well, it was old. Now on my OS X 10.3 machine, SVN 220 has some changes for
the
UNALIGNED_OK=0 case, test should still pass string_matching.
You could perform a test by hand, since it seems like "printhdr" isn't working,
and
you might get more info.
e.g.,
./xdelta3-debug -s xdelta3.h xdelta3.c OUT
./xdelta3-debug printhdr OUT # should print to stdout
./xdelta3-debug printhdr OUT PRINT # should print to PRINT
This might shed some light...
Original comment by josh.mac...@gmail.com
on 15 Dec 2007 at 12:25
Original comment by josh.mac...@gmail.com
on 15 Dec 2007 at 1:56
The printhdr commands segfault... I've attached the output of strace.
Original comment by thez...@gmail.com
on 16 Dec 2007 at 4:37
Attachments:
I can't get much from the strace output. If you were to build "xdelta3-debug"
and
run it under gdb, and show me the stacktrace, that would be best:
# gdb ./xdelta3-debug
(gdb) run printhdr INPUT
....
CRASH!
Original comment by josh.mac...@gmail.com
on 16 Dec 2007 at 7:10
That doesn't really help... I've never gotten gdb to work correctly on the
device.
All the output I get from the stacktrace is:
#0 0x00000003 in ?? ()
Original comment by thez...@gmail.com
on 17 Dec 2007 at 1:23
Original comment by josh.mac...@gmail.com
on 17 Dec 2007 at 2:31
I'd love to know if this is still a problem in the 3.0u release which I just
made.
Original comment by josh.mac...@gmail.com
on 13 Sep 2008 at 1:23
New test results for the 3.0u release:
# ./xdelta3 test
[...]
xdelta3: testing force_behavior... success
xdelta3: testing stdout_behavior... success
xdelta3: testing no_output... success
xdelta3: testing command_line_arguments................ success
xdelta3: testing recode_command...xdelta3: unrecognized window indicator bits
set:
XD3_INVALID_INPUT
xdelta3: printhdr command: xdelta3 printhdr -f /tmp/xdtest.copy.17 /tmp/
xdtest.recon2.17
failed: printhdr cmd failed: XD3_INTERNAL
# ./xdelta3 -s xdelta3.c xdelta3.h OUT
# ./xdelta3 printhdr OUT
xdelta3: unrecognized window indicator bits set: XD3_INVALID_INPUT
Original comment by thez...@gmail.com
on 15 Sep 2008 at 10:20
Thanks, this should help track it down.
Original comment by josh.mac...@gmail.com
on 15 Sep 2008 at 12:10
Can you upload the file "OUT" from the last printhdr command? This should help
diagnose whether it's an encoder-side or decoder-side issue. I've been looking
for
issues in the code but nothing sticks out.
Original comment by josh.mac...@gmail.com
on 17 Sep 2008 at 12:27
Would you try with SVN 277? I think a signed-enum with sign-extension issue
may have
cropped up.
See related report in issue 47.
Original comment by josh.mac...@gmail.com
on 17 Sep 2008 at 3:34
Ok, here's the OUT file. SVN 277 doesn't change the errors I get.
Original comment by thez...@gmail.com
on 17 Sep 2008 at 7:31
Attachments:
The file looks good, I was able to print it and reconstruct a copy of xdelta3.h
using
OUT. I will try to instrument the code with more debugging info. Thanks for
helping.
Original comment by josh.mac...@gmail.com
on 17 Sep 2008 at 12:36
How about now?
Original comment by josh.mac...@gmail.com
on 28 Oct 2009 at 3:46
Same thing:
# su -c'./xdelta3 test' nobody
Xdelta version 3.0w, Copyright (C) 2007, 2008, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
UNALIGNED_OK=0
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(void*)=4
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t.....................................
success
xdelta3: testing
encode_decode_uint64_t..........................................................
.........
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching... success
xdelta3: testing choose_instruction... success
xdelta3: testing identical_behavior... success
xdelta3: testing in_memory... success
xdelta3: testing iopt_flush_instructions... success
xdelta3: testing source_cksum_offset... success
xdelta3: testing decompress_single_bit_error.... success
xdelta3: testing decompress_single_bit_error (XD3_ADLER32).... success
xdelta3: testing decompress_single_bit_error (XD3_SEC_FGK).... success
xdelta3: testing decompress_single_bit_error (XD3_SEC_DJW).... success
xdelta3: testing force_behavior... success
xdelta3: testing stdout_behavior... success
xdelta3: testing no_output... success
xdelta3: testing command_line_arguments................ success
xdelta3: testing source_decompression... success
xdelta3: testing externally_compressed_io......sh: bzip2: command not found
bzip2=0...sh: compress: command not found
compress=0 success
xdelta3: testing recode_command...xdelta3: unrecognized window indicator bits
set:
XD3_INVALID_INPUT
xdelta3: printhdr command: ./xdelta3 printhdr -f /tmp/xdtest.copy.18 /tmp/
xdtest.recon2.18
failed: printhdr cmd failed: XD3_INTERNAL
# ./xdelta3 -s xdelta3.c xdelta3.h OUT
# ./xdelta3 printhdr OUT
xdelta3: unrecognized window indicator bits set: XD3_INVALID_INPUT
Original comment by thez...@gmail.com
on 28 Oct 2009 at 9:19
Attachments:
Original issue reported on code.google.com by
thez...@gmail.com
on 29 Nov 2007 at 4:43