search

jmacd / xdelta

open-source binary diff, delta/differential compression tools, VCDIFF/RFC 3284 delta compression
http://xdelta.org
1.12k stars 187 forks source link

Problems compiling xdelta3 under solaris 10 using gcc 3.4.3 #41

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
After installing xdelta3 on my mac running os x 10.4 I thought it would be
no bother installing it under Solaris 10. However it refused to build, but
messing about and getting it compiled it dumps core while testing:-

   ultra45(fergus)% make
   gcc -O3 -Wall -Wshadow xdelta3.c -lm -o xdelta3 \
                 -DXD3_DEBUG=0 \
                 -DXD3_USE_LARGEFILE64=1 \
                 -DREGRESSION_TEST=1 \
                 -DSECONDARY_DJW=1 \
                 -DSECONDARY_FGK=1 \
                 -DXD3_MAIN=1 \
                 -DXD3_POSIX=1
   In file included from xdelta3.c:269:
   xdelta3.h:106: error: syntax error before "uint8_t"
   xdelta3.h:106: warning: useless keyword or type name in empty declaration
   xdelta3.h:106: warning: empty declaration
   xdelta3.h:107: error: syntax error before "uint16_t"
   xdelta3.h:107: warning: useless keyword or type name in empty declaration
   xdelta3.h:107: warning: empty declaration
   xdelta3.h:110: error: syntax error before "uint32_t"
   xdelta3.h:110: warning: useless keyword or type name in empty declaration
   xdelta3.h:110: warning: empty declaration
   xdelta3.h:113: error: syntax error before "uint64_t"
   xdelta3.h:113: warning: useless keyword or type name in empty declaration
   xdelta3.h:113: warning: empty declaration
   make: *** [xdelta3] Error 1

As described above I messed with xdelta3.h as follows adding the 
section within _SOLARIS

   #define _SOLARIS
   #ifdef _SOLARIS
   typedef unsigned int       uint;
   typedef signed int         ssize_t;
   typedef unsigned int       usize_t;
   typedef unsigned char      uint8_t;
   typedef unsigned short     uint16_t;
   /* typedef unsigned long      uint32_t; */
   typedef unsigned long long uint64_t;
   #else

   #ifndef _WIN32
   typedef unsigned int       usize_t;
   typedef u_int8_t           uint8_t;
   typedef u_int16_t          uint16_t;

   #ifndef __uint32_t_defined  /* Note: Cygwin compat */
   typedef u_int32_t       uint32_t;
   #endif

   typedef u_int64_t       uint64_t;
   #else
   #define WIN32_LEAN_AND_MEAN
   #include <windows.h>
   #define inline
   typedef unsigned int   uint;
   typedef signed int     ssize_t;
   typedef unsigned int   usize_t;
   typedef unsigned char  uint8_t;
   typedef unsigned short uint16_t;
   typedef unsigned long  uint32_t;
   typedef ULONGLONG      uint64_t;
   #endif
   #endif

which allowed it to compile. I then ran its test mode:-
   ultra45(fergus)% ./xdelta3 test
   xdelta3: testing random_numbers... success
   xdelta3: testing decode_integer_end_of_input... success
   xdelta3: testing decode_integer_overflow... success
   xdelta3: testing
encode_decode_uint32_t..................................... success
   xdelta3: testing
encode_decode_uint64_t..........................................................
.........
success
   xdelta3: testing usize_t_overflow... success
   xdelta3: testing address_cache... success
   xdelta3: testing string_matching...Bus Error (core dumped)

Any ideas or help. I am running the following version of gcc.

   ultra45(fergus)% gcc -v
   Reading specs from /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/specs
   Configured with:
/gates/sfw10/builds/sfw10-gate/usr/src/cmd/gcc/gcc-3.4.3/configure
--prefix=/usr/sfw --with-as=/usr/sfw/bin/gas --with-gnu-as
--with-ld=/usr/ccs/bin/ld --without-gnu-ld --enable-languages=c,c++
--enable-shared
   Thread model: posix
   gcc version 3.4.3 (csl-sol210-3_4-branch+sol_rpath)

I repeated the above using Solaris8 and gcc 3.2.??

Original issue reported on code.google.com by fer...@twig.demon.co.uk on 17 Jul 2007 at 9:48

GoogleCodeExporter commented 9 years ago 
I have had some progress with this issue. It works! 

Running xdelta3 with the -N option allows the program to run without crashing. 
I have
used this in production a few thousand times now and it has not failed. However 
I
have gone live with code which uses xdelta twice on the server side, it creates 
the
delta, applies the delta to create a temp file which is then 'diffed' against 
the
original newer file. This is quite an overhead, but I feel I cannot fully trust 
the
current version of xdelta3.

With a colleague of mine, PTadman, we attempted to locate what was going on in 
the
code. But found it too difficult to read!

Using the -v option also causes a crash, but in a different area of the program.

Original comment by fer...@twig.demon.co.uk on 5 Nov 2007 at 6:12

GoogleCodeExporter commented 9 years ago 
The -v option may be causing crashes due to printf thinking some integer type 
is a
different size than it really is.

I wish you could provide more info on the crashes you see.  I am not receiving 
bug
reports for x86 platforms these days.  3.0r fixes a bug with "-S djw", but 
otherwise
I've been working on performance improvements. I feel that you're having
Solaris-specific issues, and I would like them fixed. I should get a 64bit 
platform
of some sort to test on..

I have thought about adding an extra safety mode for encoding, where it would
"virtually" decode the result as the data is encoded, to verify during encoding 
that
the delta is correct w/o the extra step you mention.

Original comment by josh.mac...@gmail.com on 5 Nov 2007 at 2:29

GoogleCodeExporter commented 9 years ago 
Ok, hope this helps. Let me know if there is any more I can do; but I cant send 
core 
files. The test case was a copy of the GPL with and without a s/the/hte/g. So 
repeating a working diff again without the -N option we get a core dump.

ultra45(fergus)% xdelta3 -e -s gpl-not.html gpl.html diff.out
 Bus Error (core dumped)
ultra45(fergus)% gdb /usr/local/bin/xdelta3 core
 GNU gdb 6.6
 Copyright (C) 2006 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "sparc-sun-solaris2.10"...
 Reading symbols from /lib/libm.so.2...done.
 Loaded symbols for /lib/libm.so.2
 Reading symbols from /lib/libc.so.1...done.
 Loaded symbols for /lib/libc.so.1
 Reading symbols from /platform/SUNW,A70/lib/libc_psr.so.1...done.
 Loaded symbols for /platform/SUNW,A70/lib/libc_psr.so.1
 Reading symbols from /lib/ld.so.1...done.
 Loaded symbols for /lib/ld.so.1
 Core was generated by `xdelta3 -e -s gpl-not.html gpl.html diff.out'.
 Program terminated with signal 10, Bus error.
 #0  0x00024660 in xd3_scksum (
     seg=0x5a232 "These Terms to Your New Programs\n    
</a></span></dt></dl></div><div class=\"sect1\" lang=\"en\"><div 
class=\"titlepage\"><div><div><h2 class=\"title\" style=\"clear: both\"><a 
name=\"gpl-1\"></a>Preamble</h2></d"..., ln=4)
     at xdelta3.c:1637
 1637    xdelta3.c: No such file or directory.
         in xdelta3.c
 (gdb) where
 #0  0x00024660 in xd3_scksum (
     seg=0x5a232 "These Terms to Your New Programs\n    
</a></span></dt></dl></div><div class=\"sect1\" lang=\"en\"><div 
class=\"titlepage\"><div><div><h2 class=\"title\" style=\"clear: both\"><a 
name=\"gpl-1\"></a>Preamble</h2></d"..., ln=4)
     at xdelta3.c:1637
 #1  0x000247e4 in xd3_string_match_fastest (stream=0xffbfec50) at xdelta3.c:4876
 #2  0x000214ac in xd3_encode_input (stream=0xffbfec50) at xdelta3.c:3720
 #3  0x0002e4fc in main_input (cmd=CMD_ENCODE, ifile=0xffbff0b8, ofile=0xffbff080, 
sfile=0xffbff048) at xdelta3-main.h:2571
 #4  0x00030254 in main (argc=2, argv=0xffbff17c) at xdelta3-main.h:3221
 (gdb) quit
ultra45(fergus)%

Than using the -v option.

ultra45(fergus)% xdelta3 -v -e -s gpl-not.html gpl.html diff.out
 Segmentation Fault (core dumped)
ultra45(fergus)%
ultra45(fergus)% gdb /usr/local/bin/xdelta3 core
 GNU gdb 6.6
 Copyright (C) 2006 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "sparc-sun-solaris2.10"...
 Reading symbols from /lib/libm.so.2...done.
 Loaded symbols for /lib/libm.so.2
 Reading symbols from /lib/libc.so.1...done.
 Loaded symbols for /lib/libc.so.1
 Reading symbols from /platform/SUNW,A70/lib/libc_psr.so.1...done.
 Loaded symbols for /platform/SUNW,A70/lib/libc_psr.so.1
 Reading symbols from /lib/ld.so.1...done.
 Loaded symbols for /lib/ld.so.1
 Core was generated by `xdelta3 -v -e -s gpl-not.html gpl.html diff.out'.
 Program terminated with signal 11, Segmentation fault.
 #0  0xff1b0c90 in strlen () from /lib/libc.so.1
 (gdb) where
 #0  0xff1b0c90 in strlen () from /lib/libc.so.1
 #1  0xff216258 in _ndoprnt () from /lib/libc.so.1
 #2  0xff2185e4 in sprintf () from /lib/libc.so.1
 #3  0x0002730c in main_format_bcnt (r=24, buf=0x4f8f8 "qu ") at xdelta3-main.h:540
 #4  0x0002cd90 in main_set_source (stream=0x0, cmd=4, sfile=0xffbff040, 
source=0xffbfeb70) at xdelta3-main.h:2153
 #5  0x0002e194 in main_input (cmd=CMD_ENCODE, ifile=0xffbff0b0, ofile=0xffbff078, 
sfile=0xffbff040) at xdelta3-main.h:2503
 #6  0x00030254 in main (argc=2, argv=0xffbff178) at xdelta3-main.h:3221
 (gdb)

Original comment by fer...@twig.demon.co.uk on 13 Nov 2007 at 1:41

GoogleCodeExporter commented 9 years ago 
3.0t might improve this situation. Please let me know.

Original comment by josh.mac...@gmail.com on 10 Dec 2007 at 6:12

GoogleCodeExporter commented 9 years ago 
Hi Josh,

The program version 3.0t was a lot easier to compile and test. I only
had to define the $CC variable inside the Makefile. However the program
test still failed, as shown below, however my -N workaround no longer
seems to get me past the failure. So I am sticking to the older version.

Please let me know if I can do anything more to help. 

./xdelta3-debug test
Xdelta version 3.0t, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=1
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=1
XD3_ENCODER=1
XD3_POSIX=0
XD3_STDIO=1
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... 
success
xdelta3: testing
encode_decode_uint64_t..........................................................
.........
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing address_cache (XD3_ALT_CODE_TABLE)... success
xdelta3: testing string_matching...Bus Error (core dumped)

ultra45(fergus)% gdb ./xdelta3-debug core
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.10"...
Reading symbols from /lib/libm.so.2...done.
Loaded symbols for /lib/libm.so.2
Reading symbols from /lib/libc.so.1...done.
Loaded symbols for /lib/libc.so.1
Reading symbols from /platform/SUNW,A70/lib/libc_psr.so.1...done.
Loaded symbols for /platform/SUNW,A70/lib/libc_psr.so.1
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
Core was generated by `./xdelta3-debug test'.
Program terminated with signal 10, Bus error.
#0  0x00011ed0 in xd3_string_match_soft (stream=0xffbfd458) at xdelta3.c:5223
5223          if (DO_SMALL) { SMALL_CKSUM_UPDATE (scksum, inp, SLOOK); }
(gdb) quit

ultra45(fergus)%

Original comment by fer...@twig.demon.co.uk on 16 Dec 2007 at 10:39

GoogleCodeExporter commented 9 years ago 
I have a post-3.0t that should address this problem -- it was an error found on
ARM/Xscale, and it's fixed but not all tests are passing on that platform. I'll 
post
here when 3.0u is ready.  Thanks.

Original comment by josh.mac...@gmail.com on 17 Dec 2007 at 2:16

GoogleCodeExporter commented 9 years ago 
What is a "post-3.0t" version?
Does this mean there is a version somewhere I download and try?

PS. My email has changed to fergus-at-twig.me.uk, however google seems 
incapable of 
dealing with folk changing email addresses!

Original comment by fer...@twig.demon.co.uk on 4 Aug 2008 at 12:39

GoogleCodeExporter commented 9 years ago 
Josh,

As per your suggestion I download todays cut (6-aug-2008) from cvs, comnpiled 
and 
ran its test mode. Test mode failed. Here is a log of the results along with 
other 
misc information. Let me know if there is more I can do.

fergus@ultra45: dir
total 1520
drwxr-xr-x   4 fergus   athene2     1024 Aug  6 17:23 ./
drwxrwxr-x   6 fergus   athene2     1024 Aug  6 15:55 ../
-rw-r--r--   1 fergus   athene2     6148 Aug  6 15:16 .DS_Store
-rw-r--r--   1 fergus   athene2       82 Aug  6 15:16 ._.DS_Store
-rw-r--r--   1 fergus   athene2    18009 Feb 11  2007 COPYING
-rw-r--r--   1 fergus   athene2     7705 Aug  6 16:14 Makefile
-rw-r--r--   1 fergus   athene2     6513 Jun 20 05:06 Makefile.mingw
-rw-r--r--   1 fergus   athene2     1067 Nov  9  2007 README
-rw-r--r--   1 fergus   athene2     3600 Feb 11  2007 badcopy.c
-rw-r--r--   1 fergus   athene2     4245 Dec 16  2006 badcopy.vcproj
-rw-r--r--   1 fergus   athene2    60706 Feb 11  2007 draft-korn-vcdiff.txt
drwxr-xr-x   2 fergus   athene2      512 Aug  6 15:11 examples/
-rw-r--r--   1 fergus   athene2     2096 Jul 19 17:16 index.html
-rw-r--r--   1 fergus   athene2     1105 Feb 11  2007 linkxd3lib.c
-rw-r--r--   1 fergus   athene2      255 Mar  6  2007 plot.sh
-rw-rw-r--   1 fergus   athene2      832 Aug  6 17:22 r.r
-rw-r--r--   1 fergus   athene2    36315 Feb 11  2007 rcs_junk.cc
-rw-r--r--   1 fergus   athene2     1067 Mar 25  2007 readme.txt
-rw-r--r--   1 fergus   athene2     2566 Mar 31 04:03 setup.py
drwxr-xr-x   2 fergus   athene2      512 Aug  6 15:13 testing/
-rw-r--r--   1 fergus   athene2     4222 Nov 11  2007 xdelta3-cfgs.h
-rw-r--r--   1 fergus   athene2    29880 Dec 27  2007 xdelta3-decode.h
-rw-r--r--   1 fergus   athene2    47036 Dec 14  2007 xdelta3-djw.h
-rw-r--r--   1 fergus   athene2    22155 Oct 29  2007 xdelta3-fgk.h
-rw-r--r--   1 fergus   athene2     7236 Dec 15  2007 xdelta3-hash.h
-rw-r--r--   1 fergus   athene2     8763 Nov 13  2007 xdelta3-list.h
-rw-r--r--   1 fergus   athene2   107331 Jul  7 04:11 xdelta3-main.h
-rw-r--r--   1 fergus   athene2    11876 Jul  7 04:11 xdelta3-merge.h
-rw-r--r--   1 fergus   athene2     2345 Feb 13  2007 xdelta3-python.h
-rw-r--r--   1 fergus   athene2    33420 Mar 31 04:03 xdelta3-regtest.py
-rw-r--r--   1 fergus   athene2     7172 Dec  2  2007 xdelta3-second.h
-rw-r--r--   1 fergus   athene2    75399 Jul  3 13:59 xdelta3-test.h
-rw-r--r--   1 fergus   athene2     4823 Mar 31 04:03 xdelta3-test.py
-rw-r--r--   1 fergus   athene2   154167 Jul 10 05:10 xdelta3.c
-rw-r--r--   1 fergus   athene2    45508 Jul  3 13:59 xdelta3.h
-rw-r--r--   1 fergus   athene2     2018 Dec  7  2007 xdelta3.prj
-rw-r--r--   1 fergus   athene2     2203 Feb 13  2007 xdelta3.swig
-rw-r--r--   1 fergus   athene2     4938 Mar 26  2007 xdelta3.vcproj
-rw-r--r--   1 fergus   athene2      381 Mar 25  2007 xdelta3.wxi
-rw-r--r--   1 fergus   athene2     4771 Dec  7  2007 xdelta3.wxs
fergus@ultra45: make
gcc -g -Wall -Wshadow -fno-builtin xdelta3.c -lm -o xdelta3-debug \
                -DGENERIC_ENCODE_TABLES=1 \
                -DREGRESSION_TEST=1 \
                -DSECONDARY_DJW=1 \
                -DSECONDARY_FGK=1 \
                -DXD3_DEBUG=1 \
                -DXD3_MAIN=1 \
                -DXD3_STDIO=1 \
                -DXD3_USE_LARGEFILE64=1
gcc -Wall -Wshadow -fno-builtin -O3 xdelta3.c -lm -o xdelta3 \
              -DGENERIC_ENCODE_TABLES=0 \
              -DREGRESSION_TEST=1 \
              -DSECONDARY_DJW=1 \
              -DSECONDARY_FGK=1 \
              -DXD3_DEBUG=0 \
              -DXD3_MAIN=1 \
              -DXD3_POSIX=1 \
              -DXD3_USE_LARGEFILE64=1
fergus@ultra45: ./xdelta3-debug test
Xdelta version 3.0t, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=1
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
UNALIGNED_OK=0
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=1
XD3_ENCODER=1
XD3_POSIX=0
XD3_STDIO=1
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(void*)=4
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... failed: random distribution looks broken: 
XD3_INTERNAL
fergus@ultra45: 
fergus@ultra45: 
fergus@ultra45: uname -a
SunOS ultra45 5.10 Generic_118833-24 sun4u sparc SUNW,A70
fergus@ultra45: gcc --version
gcc (GCC) 3.4.6
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Original comment by fer...@twig.demon.co.uk on 6 Aug 2008 at 4:48

GoogleCodeExporter commented 9 years ago 
Via e-mail, after the random issue was solved:

xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... 
success
xdelta3: testing
encode_decode_uint64_t..........................................................
.........
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing address_cache (XD3_ALT_CODE_TABLE)... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10
C12/8@qd failed: wrong result: XD3_INTERNAL

This is the sign of a checksum error, related to endianness.  I thought I had 
fixed
this in December of last year, but apparently not.  At this time I do not have a
machine to test this with, but I will investigate.  (I was in the middle of 
working
on Issue 36 and couldn't take my attention away from that, but now I've reached 
a
good stopping point in that work.)

Original comment by josh.mac...@gmail.com on 7 Sep 2008 at 9:56

GoogleCodeExporter commented 9 years ago 
Actually, this looks like an issue with printf(), not endian-ness!

See how it says expected "R/10 C12/8@10" but it got "R0/10 C12/8@qd"?  The "qd" 
is
part of a format specifier for the address portion of the copy instruction, 
which
should be "10" but instead it prints "qd".

I've upgraded from the deprecated "q" specifier to "ll", which I believe is 
portable,
SVN 273.

Original comment by josh.mac...@gmail.com on 7 Sep 2008 at 3:59