search

ECToo / xdelta

Automatically exported from code.google.com/p/xdelta
0 stars 0 forks source link

xdelta3 test fails (valgrind data) #51

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Running xdelta3 test with xdelta3 3.0s results in :

xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing
encode_decode_uint32_t..................................... success
xdelta3: testing
encode_decode_uint64_t..........................................................
.........
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching... success
xdelta3: testing choose_instruction... success
xdelta3: testing identical_behavior... failed: getblk returned short block:
XD3_INTERNAL

This is with Linux 2.6.24-rc3, glibc 2.7, gcc 4.3 trunk.

Original issue reported on code.google.com by ismail.d...@gmail.com on 26 Nov 2007 at 5:48

GoogleCodeExporter commented 9 years ago 
I get a different error on Windows XP, using the supplied xdelta30s.exe binary:

xdelta3: testing decompress_single_bit_error (XD3_SEC_DJW)...non-failures 17;
expected 18 failed: incorrect: XD3_INTERNAL

Could this be what corrupted my deltas? (see issue #50)

Also, the 30s tests took a LOT longer than the tests on 30q.

Original comment by nicolas....@gmail.com on 26 Nov 2007 at 5:25

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
The "expected non-failures" test failure is very odd.  Basically the test is
producing different values in optimized mode and debug mode, meaning however I 
set
the "expected non-failures" value it will be wrong in one of the builds.  This 
seems
like it could be related to the compiler optimizing away some code or a 
compiler bug.
 I will need to test the current code w/ gcc 4.3 to see if I can reproduce the
"getblk returned short block: XD3_INTERNAL" problem and also whether the newer
compiler changes the expected non-failures result. Weird.

If you'd like to help out, you could get the current code from subversion and 
build
w/ gcc 4.3 and see what "xdelta3 test" and "xdelta3-debug test" produce.

Lately I've been developing/testing on Cygwin with gcc 3.4.4, I'm embarrased to 
admit.

Original comment by josh.mac...@gmail.com on 27 Nov 2007 at 8:00

GoogleCodeExporter commented 9 years ago 
Ok good news, the reason that the test fails is because I modifed Makefile to 
add
-fomit-frame-pointer, so it looks like this :

cc -fomit-frame-pointer -O3 -Wall -Wshadow -fno-builtin xdelta3.c -lm -o 
xdelta3 \
              -DGENERIC_ENCODE_TABLES=0 \
              -DREGRESSION_TEST=1 \
              -DSECONDARY_DJW=1 \
              -DSECONDARY_FGK=1 \
              -DUNALIGNED_OK=1 \
              -DXD3_DEBUG=0 \
              -DXD3_MAIN=1 \
              -DXD3_POSIX=1 \
              -DXD3_USE_LARGEFILE64=1

And this makes regression tests fail, otherwise they pass. This might be a gcc 
bug or
not, I would need your help to decide.

Original comment by ismail.d...@gmail.com on 27 Nov 2007 at 9:12

GoogleCodeExporter commented 9 years ago 
xd3_blksize_div mentions __udivdi3 and I somehow think
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=32044 might be relevant.

Original comment by ismail.d...@gmail.com on 27 Nov 2007 at 11:56

GoogleCodeExporter commented 9 years ago 
That's an interesting bug report (I'm surprised gcc can translate a 
subtract-loop
into a divide), however it seems to be about a mis-optimization (and trouble 
building
the kernel, which implements its own libgcc routines).

The reference to __udivdi3 is not problematic.  The changes in 3.0s were in 
fact to
reduce the number of calls to __udivdi3 (e.g., replace a pair of division and 
mod
operations by a single divide/multiply/subtract).  Also in 3.0s, some code to 
avoid
division entirely when the divisor is a power-of-two, so that in the typical 
case,
for power-of-two source block sizes (the value of -B) there is no division at 
all.

Unfortunately, there's still a bug in xdelta's test which seems related to gcc.

Original comment by josh.mac...@gmail.com on 28 Nov 2007 at 6:40

GoogleCodeExporter commented 9 years ago 
If I can help in anyway, please let me know. I tried to compare assembly output
bewteen -fomit-frame-pointer and the normal flags and assembly difference is 
too big
for me to make sense of it.

Original comment by ismail.d...@gmail.com on 28 Nov 2007 at 8:14

GoogleCodeExporter commented 9 years ago

Original comment by josh.mac...@gmail.com on 30 Nov 2007 at 5:20

GoogleCodeExporter commented 9 years ago 
Pleased to report this is fixed in SVN 205, will release soon.

Original comment by josh.mac...@gmail.com on 2 Dec 2007 at 3:14

GoogleCodeExporter commented 9 years ago 
Sadly cant confirm fix with latest SVN (Just added -fomit-frame-pointer to 
CFLAGS) :

[~/xdelta-read-only/xdelta3]> svn info
Path: .
URL: http://xdelta.googlecode.com/svn/trunk/xdelta3
Repository Root: http://xdelta.googlecode.com/svn
Repository UUID: a3eca27d-f21b-0410-9b4a-6511e771f64e
Revision: 205
[...]

[~/xdelta-read-only/xdelta3]> ./xdelta3 test
Xdelta version 3.0s, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... 
success
xdelta3: testing
encode_decode_uint64_t..........................................................
.........
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching... success
xdelta3: testing choose_instruction... success
xdelta3: testing identical_behavior... failed: getblk returned short block: 
XD3_INTERNAL

Original comment by ismail.d...@gmail.com on 2 Dec 2007 at 4:25

GoogleCodeExporter commented 9 years ago 
Here is the valgrind output, there might be another bug hiding there :

xdelta3: testing identical_behavior...==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x8065056: test_identical_behavior (xdelta3.h:1192)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x806507D: test_identical_behavior (xdelta3.h:1192)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x8048F1E: xd3_blksize_div (xdelta3.h:1192)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x8048F2F: xd3_blksize_div (xdelta3.h:1192)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x8048F43: xd3_blksize_div (xdelta3.h:1195)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x805F072: xd3_source_extend_match (xdelta3.h:1192)
==8871==    by 0x3E7: ???
==8871==    by 0x2BFA3E0B: ???
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x805F143: xd3_source_extend_match (xdelta3.h:1192)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x805F10F: xd3_source_extend_match (xdelta3.c:2499)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x805F111: xd3_source_extend_match (xdelta3.c:2499)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x805F163: xd3_source_extend_match (xdelta3.c:2499)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x805F0F8: xd3_source_extend_match (xdelta3.c:2496)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x805F106: xd3_source_extend_match (xdelta3.c:2496)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x805F1AC: xd3_source_extend_match (xdelta3.c:2523)
==8871==
==8871== Use of uninitialised value of size 4
==8871==    at 0x805F211: xd3_source_extend_match (xdelta3.c:4393)
==8871==
==8871== Use of uninitialised value of size 4
==8871==    at 0x805F220: xd3_source_extend_match (xdelta3.c:4393)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x805F0E6: xd3_source_extend_match (xdelta3.c:4488)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x805F1E7: xd3_source_extend_match (xdelta3.c:4506)
==8871==
==8871== Conditional jump or move depends on uninitialised value(s)
==8871==    at 0x805F1FA: xd3_source_extend_match (xdelta3.c:4393)
==8871==
==8871== Invalid read of size 1
==8871==    at 0x805F211: xd3_source_extend_match (xdelta3.c:4393)
==8871==  Address 0xBEDB504C is not stack'd, malloc'd or (recently) free'd
==8871==
==8871== Process terminating with default action of signal 11 (SIGSEGV)
==8871==  Access not within mapped region at address 0xBEDB504C
==8871==    at 0x805F211: xd3_source_extend_match (xdelta3.c:4393)

Original comment by ismail.d...@gmail.com on 2 Dec 2007 at 11:12

GoogleCodeExporter commented 9 years ago 
Very interesting!  I will investigate.

Original comment by josh.mac...@gmail.com on 2 Dec 2007 at 11:58

GoogleCodeExporter commented 9 years ago 
By the way, I should have reported that SVN 205 fixes the error first reported 
in
this thread:

xdelta3: testing decompress_single_bit_error (XD3_SEC_DJW)...non-failures 17;
expected 18 failed: incorrect: XD3_INTERNAL

Uninitialized data strikes again. Thanks for the valgrind data.

Original comment by josh.mac...@gmail.com on 2 Dec 2007 at 7:14

GoogleCodeExporter commented 9 years ago 
I found a bug in xd3_set_source() that should solve this problem, SVN 207.  
Please
let me know. Thanks!

Original comment by josh.mac...@gmail.com on 5 Dec 2007 at 3:54

GoogleCodeExporter commented 9 years ago 
Confirming the fix! Very nice work, waiting for the new release :-)

Original comment by ismail.d...@gmail.com on 5 Dec 2007 at 8:44

GoogleCodeExporter commented 9 years ago 
Spoke too early, same error, somewhere down the regression tests :

[~/xdelta3]> ./xdelta3 test
Xdelta version 3.0s, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... 
success
xdelta3: testing
encode_decode_uint64_t..........................................................
.........
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching... success
xdelta3: testing choose_instruction... success
xdelta3: testing identical_behavior... success
xdelta3: testing in_memory... success
xdelta3: testing iopt_flush_instructions... success
xdelta3: testing source_cksum_offset... success
xdelta3: testing decompress_single_bit_error.... success
xdelta3: testing decompress_single_bit_error (XD3_ADLER32).... success
xdelta3: testing decompress_single_bit_error (XD3_SEC_FGK).... success
xdelta3: testing decompress_single_bit_error (XD3_SEC_DJW).... success
xdelta3: testing secondary_huff...
...2.000....7.219....8.188....4.891....4.891....7.022....8.039....2.017....7.937
....7.408....2.618....
...2.160....7.312....8.250....4.953....4.953....7.123....8.194....2.023....6.953
....6.418....2.448....
...2.240....7.344....8.281....5.008....5.008....7.191....8.281....2.025....6.375
....5.831....1.843....
...2.240....7.375....8.281....5.048....5.048....7.241....8.353....2.025....5.990
....5.436....1.679....
...2.320....7.375....8.312....5.053....5.053....7.275....8.434....2.027....5.671
....5.117....1.654....
...2.320....7.375....8.312....5.091....5.091....7.331....8.525....2.027....5.418
....4.869....1.567....
...2.320....7.406....8.344....5.131....5.131....7.350....8.591....2.027....5.219
....4.654....1.352....
...2.400....7.406....8.344....5.144....5.144....7.403....8.658....2.027....5.040
....4.471....1.352....
success
xdelta3: testing secondary_fgk...
...1.120....11.000....15.000....4.980....4.980....7.266....8.450....2.256....8.0
08....7.409....2.618....
success
xdelta3: testing force_behavior... success
xdelta3: testing stdout_behavior... success
xdelta3: testing no_output... success
xdelta3: testing command_line_arguments........xdelta3: test encode with size 
ratio
0.0707, expected > -0.1500
 failed: strange encoding: XD3_INTERNAL

Valgrind logs will follow.

Original comment by ismail.d...@gmail.com on 5 Dec 2007 at 8:49

GoogleCodeExporter commented 9 years ago 
No valgrind errors this time, valgrind reports :

==11808== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 15 from 1)

Original comment by ismail.d...@gmail.com on 5 Dec 2007 at 10:02

GoogleCodeExporter commented 9 years ago 
The new error will be easy to fix.  I actually tightened the range checks in 
that
test while I was debugging the errors fixed in SVN 205.  Stay tuned.

Original comment by josh.mac...@gmail.com on 5 Dec 2007 at 12:15

GoogleCodeExporter commented 9 years ago 
Regression tests passes with revision 208, I am running through valgrind again 
to see
if any error slipped in.

Thank you for your response and fast fix.

Original comment by ismail.d...@gmail.com on 6 Dec 2007 at 6:49

GoogleCodeExporter commented 9 years ago 
and valgrind passes too :

==14919== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 15 from 1)

It just took 2 hours but well its worth it ;)

Really waiting for a new release now :-)

Original comment by ismail.d...@gmail.com on 6 Dec 2007 at 8:51

GoogleCodeExporter commented 9 years ago

Original comment by josh.mac...@gmail.com on 7 Dec 2007 at 5:31