search

ECToo / xdelta

Automatically exported from code.google.com/p/xdelta
0 stars 0 forks source link

xdelta3: printdelta segfault (arm/xscale) #52

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. Run xdelta3 test on a arm/xscale system (familiar linux)

What is the expected output? What do you see instead?
Test output is:

root@familiar:/tmp# ./xdelta3 test
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing 
encode_decode_uint32_t..................................... success
xdelta3: testing 
encode_decode_uint64_t..........................................................
.........
 success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/
10 C14/6@12 failed: wrong result: XD3_INTERNAL

What version of the product are you using? On what operating system?
I tried xdelta3 for ARM from debian sid and lenny (q & s) as well as my 
own compiled binaries (k & s).

Should I still be able to use xdelta3 normally, even though this test 
fails?

Original issue reported on code.google.com by thez...@gmail.com on 29 Nov 2007 at 4:43

GoogleCodeExporter commented 9 years ago 
I'm a little worried by this.  I haven't seen it on any other platform.  Can 
you tell
me the compiler version?  Issue 51 is related to a compiler bug as well.  I am 
not
familiar with the arm/xscale platform.  What is xscale?

Original comment by josh.mac...@gmail.com on 29 Nov 2007 at 4:55

GoogleCodeExporter commented 9 years ago 
XScale is an ARM processor (http://en.wikipedia.org/wiki/Xscale).

The compiler I used:
Configured with: /work/crosstool-0.27/build/arm-linux/gcc-3.4.1-glibc-2.3.2/gcc-
3.4.1/configure --target=arm-linux --host=i686-host_pc-linux-gnu 
--prefix=/usr/local/
arm/3.4.1 --with-headers=/usr/local/arm/3.4.1/arm-linux/include 
--with-local-prefix=/
usr/local/arm/3.4.1/arm-linux --disable-nls --enable-threads=posix --enable-
symvers=gnu --enable-__cxa_atexit --enable-languages=c,c++ --enable-shared 
--enable-
c99 --enable-long-long
Thread model: posix
gcc version 3.4.1

What the official debian binaries were compiled with I don't know. But at least 
the 
error is consistent with everything I've tried so far. I even tried with the 
compiler option I use for all my other ARM programs:
-Wall -fomit-frame-pointer -finline-functions -funroll-loops 
-falign-functions=2 -
falign-loops=2 -falign-jumps=2 -mcpu=xscale -mtune=xscale

no difference...

Compared to the expected output everything seems to be shifted by two bytes. Is 
there anything I can do to provide you with further clues for this?

Original comment by thez...@gmail.com on 29 Nov 2007 at 5:38

GoogleCodeExporter commented 9 years ago 
Can you try compiling without any optimization, and re-run the test?

Also, -fomit-frame-pointer has caused problems in issue 51.

I do a lot of development on gcc 3.4, so this seems like a real issue.

Original comment by josh.mac...@gmail.com on 30 Nov 2007 at 4:16

GoogleCodeExporter commented 9 years ago 
xdelta3 compiled with no optimizations and xdelta3-debug produce the same error 
as 
well.

Original comment by thez...@gmail.com on 30 Nov 2007 at 8:29

GoogleCodeExporter commented 9 years ago 
If you could, I'm interested in the output of "xdelta3 config".  I'm not sure 
what
I'm looking for, but I'm interested.

Original comment by josh.mac...@gmail.com on 1 Dec 2007 at 5:05

GoogleCodeExporter commented 9 years ago 
I believe this and issue 51 are caused by the same underlying issue, 
uninitialized
data. I will keep you posted.

Original comment by josh.mac...@gmail.com on 2 Dec 2007 at 7:15

GoogleCodeExporter commented 9 years ago 
Ok, here's the "xdelta3 config" output for version 3.0s:

Xdelta version 3.0s, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8

Original comment by thez...@gmail.com on 3 Dec 2007 at 8:15

GoogleCodeExporter commented 9 years ago 
Would you mind testing with SVN 207? I believe the problem is solved, there was 
a bug
introduced in 3.0s.

Original comment by josh.mac...@gmail.com on 5 Dec 2007 at 3:55

GoogleCodeExporter commented 9 years ago 
The test also fails with versions k and q as stated above.
Same thing unfortunately:

root@familiar:/tmp# ./xdelta3 test
Xdelta version 3.0s, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... 
success
xdelta3: testing 
encode_decode_uint64_t..........................................................
.........
 success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10 
C14/
6@12 failed: wrong result: XD3_INTERNAL

Original comment by thez...@gmail.com on 5 Dec 2007 at 8:08

GoogleCodeExporter commented 9 years ago 
Can you try a few more tests?  The xdelta3-32 binary compiles with a 32-bit file
offset, and its test might shed some light on this.  I've released 3.0t now, 
and this
is my highest priority bug report.

Original comment by josh.mac...@gmail.com on 7 Dec 2007 at 5:55

GoogleCodeExporter commented 9 years ago 
Ok, I can do this on monday at the earliest.

Original comment by thez...@gmail.com on 8 Dec 2007 at 1:24

GoogleCodeExporter commented 9 years ago 
Run with svn rev 210

root@familiar:/tmp# ./xdelta3-32 test
Xdelta version 3.0t, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=1
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=0
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=4
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... 
success
xdelta3: testing 
encode_decode_uint64_t..........................................................
.........
 success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10 
C14/
6@12 failed: wrong result: XD3_INTERNAL

root@familiar:/tmp# ./xdelta3 test
Xdelta version 3.0t, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... 
success
xdelta3: testing 
encode_decode_uint64_t..........................................................
.........
 success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10 
C14/
6@12 failed: wrong result: XD3_INTERNAL

Original comment by thez...@gmail.com on 10 Dec 2007 at 9:49

GoogleCodeExporter commented 9 years ago 
Thanks. I will go study the code and see if I can guess at a cause.

Original comment by josh.mac...@gmail.com on 10 Dec 2007 at 6:11

GoogleCodeExporter commented 9 years ago 
I have a theory.  ARM processors do not support unaligned access the way x86 
does. 
There is an unaligned access which may have contributed to this problem.  Could 
you
please run "xdelta3-debug test" and see if it produces the same errors?  I am 
hoping
it will crash with an assertion prior to the test failure. 

SVN 211 has a few cleanups that should not make a difference, but it would be 
best if
you could test with the latest version.

I will go about cleaning up the unaligned accesses using an memcpy for 
platforms that
do not support efficient unaligned access.

Original comment by josh.mac...@gmail.com on 13 Dec 2007 at 6:31

GoogleCodeExporter commented 9 years ago 
It produces the same output as the other tests. But then again it doesn't seem 
to 
violate any bounds. The string match result is offset by two bytes but seems ok 
otherwise.

root@familiar:/tmp# ./xdelta3-debug test
Xdelta version 3.0t, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=1
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=1
XD3_ENCODER=1
XD3_POSIX=0
XD3_STDIO=1
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... 
success
xdelta3: testing 
encode_decode_uint64_t..........................................................
.........
 success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing address_cache (XD3_ALT_CODE_TABLE)... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10 
C14/
6@12 failed: wrong result: XD3_INTERNAL

Original comment by thez...@gmail.com on 13 Dec 2007 at 8:46

GoogleCodeExporter commented 9 years ago 
It is finding a correct match, but only when it starts at 12, not at 10.  I 
believe
this is because 12 is word-aligned and 10 is not.  Let me see if I can add some
debugging (to diagnose further) and/or correct the problem (the solution 
involves
memcpy).

Original comment by josh.mac...@gmail.com on 14 Dec 2007 at 5:40

GoogleCodeExporter commented 9 years ago 
SVN 212 may solve the problem.

To be EXTRA sure, you could add -DUNALIGNED_OK=0 to CFLAGS,
but I've set things so that only __i386__ and similar defs
will set that variable by default.

Original comment by josh.mac...@gmail.com on 14 Dec 2007 at 6:37

GoogleCodeExporter commented 9 years ago 
Good job, that fixed it :)

Another failing test:

xdelta3: testing no_output...command was ./xdelta3 -q -f -e 
/tmp/xdtest.target.3 /
tmp/xdtest.noperm.3 failed: command should have not succeeded: XD3_INTERNAL

Do you think it's related to that one or should I open a new ticket?

Original comment by thez...@gmail.com on 14 Dec 2007 at 1:04

GoogleCodeExporter commented 9 years ago 
We can keep this ticket open.

The test that fails now is related to file permissions.  Are you running as 
root,
possibly?  Are there file permissions?

Original comment by josh.mac...@gmail.com on 14 Dec 2007 at 6:19

GoogleCodeExporter commented 9 years ago

Original comment by josh.mac...@gmail.com on 14 Dec 2007 at 6:40

GoogleCodeExporter commented 9 years ago 
Actually I want to keep this issue open for another reason.

Now that we know the string_matching problem is fixed, I'd like to test a more
efficient solution.  Stay tuned.

Original comment by josh.mac...@gmail.com on 14 Dec 2007 at 7:30

GoogleCodeExporter commented 9 years ago 
Argh--I just cleaned up the UNALIGNED_OK=0 case while sitting at the laundromat 
and
spill bleach on my computer (bad) on the way home!

Original comment by josh.mac...@gmail.com on 14 Dec 2007 at 10:48

GoogleCodeExporter commented 9 years ago 
Aw, shit. Hope it still works...

root is actually the only user on the device... I created a test user now and 
with 
that it works alright. Next fail:

xdelta3: testing recode_command...xdelta3: printhdr command: /tmp/xdelta3 
printhdr -
f /tmp/xdtest.copy.17 /tmp/xdtest.recon2.17 failed: printhdr cmd failed: 
XD3_INTERNAL

Original comment by thez...@gmail.com on 15 Dec 2007 at 12:04

GoogleCodeExporter commented 9 years ago 
There's a sizzling sound from the bleach on the battery contacts.  Not good.

Oh well, it was old.  Now on my OS X 10.3 machine, SVN 220 has some changes for 
the
UNALIGNED_OK=0 case, test should still pass string_matching.

You could perform a test by hand, since it seems like "printhdr" isn't working, 
and
you might get more info.

e.g.,

./xdelta3-debug -s xdelta3.h xdelta3.c OUT
./xdelta3-debug printhdr OUT  # should print to stdout
./xdelta3-debug printhdr OUT PRINT # should print to PRINT

This might shed some light...

Original comment by josh.mac...@gmail.com on 15 Dec 2007 at 12:25

GoogleCodeExporter commented 9 years ago

Original comment by josh.mac...@gmail.com on 15 Dec 2007 at 1:56

GoogleCodeExporter commented 9 years ago 
The printhdr commands segfault... I've attached the output of strace.

Original comment by thez...@gmail.com on 16 Dec 2007 at 4:37

Attachments:

GoogleCodeExporter commented 9 years ago 
I can't get much from the strace output.  If you were to build "xdelta3-debug" 
and
run it under gdb, and show me the stacktrace, that would be best:

# gdb ./xdelta3-debug
(gdb) run printhdr INPUT
....
CRASH!

Original comment by josh.mac...@gmail.com on 16 Dec 2007 at 7:10

GoogleCodeExporter commented 9 years ago 
That doesn't really help... I've never gotten gdb to work correctly on the 
device. 
All the output I get from the stacktrace is:

#0 0x00000003 in ?? ()

Original comment by thez...@gmail.com on 17 Dec 2007 at 1:23

GoogleCodeExporter commented 9 years ago

Original comment by josh.mac...@gmail.com on 17 Dec 2007 at 2:31

GoogleCodeExporter commented 9 years ago 
I'd love to know if this is still a problem in the 3.0u release which I just 
made.

Original comment by josh.mac...@gmail.com on 13 Sep 2008 at 1:23

GoogleCodeExporter commented 9 years ago 
New test results for the 3.0u release:

# ./xdelta3 test
[...]
xdelta3: testing force_behavior... success
xdelta3: testing stdout_behavior... success
xdelta3: testing no_output... success
xdelta3: testing command_line_arguments................ success
xdelta3: testing recode_command...xdelta3: unrecognized window indicator bits 
set: 
XD3_INVALID_INPUT
xdelta3: printhdr command: xdelta3 printhdr -f /tmp/xdtest.copy.17 /tmp/
xdtest.recon2.17
 failed: printhdr cmd failed: XD3_INTERNAL

# ./xdelta3 -s xdelta3.c xdelta3.h OUT
# ./xdelta3 printhdr OUT
xdelta3: unrecognized window indicator bits set: XD3_INVALID_INPUT

Original comment by thez...@gmail.com on 15 Sep 2008 at 10:20

GoogleCodeExporter commented 9 years ago 
Thanks, this should help track it down.

Original comment by josh.mac...@gmail.com on 15 Sep 2008 at 12:10

GoogleCodeExporter commented 9 years ago 
Can you upload the file "OUT" from the last printhdr command?  This should help
diagnose whether it's an encoder-side or decoder-side issue.  I've been looking 
for
issues in the code but nothing sticks out.

Original comment by josh.mac...@gmail.com on 17 Sep 2008 at 12:27

GoogleCodeExporter commented 9 years ago 
Would you try with SVN 277?  I think a signed-enum with sign-extension issue 
may have
cropped up.

See related report in issue 47.

Original comment by josh.mac...@gmail.com on 17 Sep 2008 at 3:34

GoogleCodeExporter commented 9 years ago 
Ok, here's the OUT file. SVN 277 doesn't change the errors I get.

Original comment by thez...@gmail.com on 17 Sep 2008 at 7:31

Attachments:

GoogleCodeExporter commented 9 years ago 
The file looks good, I was able to print it and reconstruct a copy of xdelta3.h 
using
OUT.  I will try to instrument the code with more debugging info.  Thanks for 
helping.

Original comment by josh.mac...@gmail.com on 17 Sep 2008 at 12:36

GoogleCodeExporter commented 9 years ago 
How about now?

Original comment by josh.mac...@gmail.com on 28 Oct 2009 at 3:46

GoogleCodeExporter commented 9 years ago 
Same thing:

# su -c'./xdelta3 test' nobody
Xdelta version 3.0w, Copyright (C) 2007, 2008, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
UNALIGNED_OK=0
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(void*)=4
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... 
success
xdelta3: testing 
encode_decode_uint64_t..........................................................
......... 
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching... success
xdelta3: testing choose_instruction... success
xdelta3: testing identical_behavior... success
xdelta3: testing in_memory... success
xdelta3: testing iopt_flush_instructions... success
xdelta3: testing source_cksum_offset... success
xdelta3: testing decompress_single_bit_error.... success
xdelta3: testing decompress_single_bit_error (XD3_ADLER32).... success
xdelta3: testing decompress_single_bit_error (XD3_SEC_FGK).... success
xdelta3: testing decompress_single_bit_error (XD3_SEC_DJW).... success
xdelta3: testing force_behavior... success
xdelta3: testing stdout_behavior... success
xdelta3: testing no_output... success
xdelta3: testing command_line_arguments................ success
xdelta3: testing source_decompression... success
xdelta3: testing externally_compressed_io......sh: bzip2: command not found
bzip2=0...sh: compress: command not found
compress=0 success
xdelta3: testing recode_command...xdelta3: unrecognized window indicator bits 
set: 
XD3_INVALID_INPUT
xdelta3: printhdr command: ./xdelta3 printhdr -f /tmp/xdtest.copy.18 /tmp/
xdtest.recon2.18
 failed: printhdr cmd failed: XD3_INTERNAL

# ./xdelta3 -s xdelta3.c xdelta3.h OUT
# ./xdelta3 printhdr OUT
xdelta3: unrecognized window indicator bits set: XD3_INVALID_INPUT

Original comment by thez...@gmail.com on 28 Oct 2009 at 9:19

Attachments: