Brotli triggers error from ASan flush unit test

[nemequ]

# mkdir build && cd build
# CFLAGS="-fsanitize=address" CXXFLAGS="-fsanitize=address" ../autogen.sh && make -j
# SQUASH_PLUGINS=plugins ./tests/flush --seed=R02S5fc65c3fbeea2c125368cfe5358f5ec9
…
/flush/brieflz/brieflz: OK
/flush/brotli/brotli: =================================================================
==9544==ERROR: AddressSanitizer: unknown-crash on address 0x611000005389 at pc 0x7f45a295fd04 bp 0x7ffe63ec49f0 sp 0x7ffe63ec49e0
WRITE of size 8 at 0x611000005389 thread T0
    #0 0x7f45a295fd03 in BROTLI_UNALIGNED_STORE64(void*, unsigned long) /home/nemequ/local/src/squash/plugins/brotli/brotli/enc/././././port.h:148
    #1 0x7f45a295fd03 in brotli::WriteBits(int, unsigned long, int*, unsigned char*) /home/nemequ/local/src/squash/plugins/brotli/brotli/enc/./write_bits.h:63
    #2 0x7f45a295fd03 in brotli::BrotliCompressor::WriteMetadata(unsigned long, unsigned char const*, bool, unsigned long*, unsigned char*) /home/nemequ/local/src/squash/plugins/brotli/brotli/enc/encode.cc:545
    #3 0x7f45a28f9c92 in squash_brotli_compress_stream /home/nemequ/local/src/squash/plugins/brotli/squash-brotli.cpp:191
    #4 0x7f45a28f9c92 in squash_brotli_process_stream /home/nemequ/local/src/squash/plugins/brotli/squash-brotli.cpp:261
    #5 0x7f45a6f344fe in squash_stream_process_internal /home/nemequ/local/src/squash/squash/stream.c:699
    #6 0x401ad7 in flush_test /home/nemequ/local/src/squash/tests/flush.c:23
    #7 0x7f45a69ff45a  (/lib64/libglib-2.0.so.0+0x6f45a)
    #8 0x7f45a69ff622  (/lib64/libglib-2.0.so.0+0x6f622)
    #9 0x7f45a69ff622  (/lib64/libglib-2.0.so.0+0x6f622)
    #10 0x7f45a69ff82d in g_test_run_suite (/lib64/libglib-2.0.so.0+0x6f82d)
    #11 0x7f45a69ff850 in g_test_run (/lib64/libglib-2.0.so.0+0x6f850)
    #12 0x401705 in main /home/nemequ/local/src/squash/tests/test-codecs.c:63
    #13 0x7f45a61e357f in __libc_start_main (/lib64/libc.so.6+0x2057f)
    #14 0x4017f8 in _start (/home/nemequ/local/src/squash/build/tests/flush+0x4017f8)

0x611000005390 is located 0 bytes to the right of 208-byte region [0x6110000052c0,0x611000005390)
allocated by thread T0 here:
    #0 0x7f45a71dba0a in malloc (/lib64/libasan.so.2+0x98a0a)
    #1 0x7f45a28fa04a in squash_brotli_stream_new /home/nemequ/local/src/squash/plugins/brotli/squash-brotli.cpp:88
    #2 0x7f45a28fa04a in squash_brotli_create_stream /home/nemequ/local/src/squash/plugins/brotli/squash-brotli.cpp:146
    #3 0x7f45a6f26dba in squash_codec_create_stream /home/nemequ/local/src/squash/squash/codec.c:519

SUMMARY: AddressSanitizer: unknown-crash /home/nemequ/local/src/squash/plugins/brotli/brotli/enc/././././port.h:148 BROTLI_UNALIGNED_STORE64(void*, unsigned long)
Shadow bytes around the buggy address:
  0x0c227fff8a20: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
  0x0c227fff8a30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c227fff8a40: 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa
  0x0c227fff8a50: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c227fff8a60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c227fff8a70: 00[00]fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8a80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8a90: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
  0x0c227fff8aa0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c227fff8ab0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8ac0: fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==9544==ABORTING

Happens on F23 x86_64, but it's probably reproducible everywhere. This may (or may not) be related to the recent issues with brotli flushing on Win64 we have been seeing in AppVeyor.

@eustas, I was hoping you might want to take a look at this. I'm not sure whether the issue is in the Squash plugin or Brotli, but at this point I think most of the plugin is code you've written so you might want to know either way. If you want I can try to put together a test case which reproduces the issue without Squash and either file a bug against Brotli or, if it's an issue in Squash, look into fixing it myself.

Basically, do you want me to consider you the maintainer of the brotli plugin for squash, or just a contributor?

[jibsen]

From a quick glance at the code, it seems like it may be storing a 64-bit value into the 6 byte buffer seal_out in squash_brotli.cpp.

[nemequ]

Yeah, that sounds right. AFAICT that would be a bug in brotli since s->remaining_out is set to 6…

[jibsen]

Yes, it looks like WriteMetadata is checking if there is 6 bytes left, and then proceeds to write uint64_t values with WriteBits (on little endian).

[nemequ]

I guess that means it probably isn't to blame for the stall on 64 bit windows :(

[jibsen]

I did a quick test locally, and increasing the size of seal_out to 16 bytes, so there is enough room for the memory writes, fixed flush crashing with the seed from the AppVeyor build you linked.

This of course isn't a fix for the actual issue, but indicates it may solve the AppVeyor build.

[eustas]

Ooops. Looks like there are some implicit requirements in encoder. Going to fix it soon.