xpress requires 4 extra bytes of output for input at multiples of 32 bytes

[nemequ]

For uncompressible data xpress always requires 4 more bytes of data than xpress_max_compressed_size says, or xpress_compress it will return 0. It doesn't actually use the extra space for output, it just refuses to compress. The issue also occurs for short but compressible data, like the first 32 bytes of enwik8, but for /dev/urandom it is every multiple of 32. Test case:

#include <stdio.h>
#include <assert.h>

#include "xpress.h"

int main(int argc, char *argv[])
{
  static uint8_t uncompressed[4096];
  static uint8_t compressed[1024 * 1024];
  size_t compressed_length;
  size_t uncompressed_length;

  assert (argc == 2);

  FILE* r = fopen (argv[1], "r");
  fread (uncompressed, 1, sizeof(uncompressed), r);
  fclose (r);

  for (uncompressed_length = 1 ;
       uncompressed_length < sizeof(uncompressed) ;
       uncompressed_length++) {
    compressed_length = xpress_max_compressed_size(uncompressed_length);
    assert(compressed_length < sizeof(compressed));
    compressed_length = xpress_compress(uncompressed, uncompressed_length,
                                        compressed, compressed_length);
    if (compressed_length == 0) {
      fprintf(stderr, "Failed at %zu\n", uncompressed_length);
      assert((compressed_length % 32) == 0);
    }
  }

  return 0;
}

[coderforlife]

I am looking into this. From what I can see, this is because there is a 4-byte (32-bit) 'flags' every 32 output symbols. When it is uncompressible, those 32 output symbols are each 1 literal byte and the flag that comes before it is "full" and it seems the next set of flags is started (an additional 4 bytes). So I have to look into what is "proper":

• Is the last flags supposed to be added even if empty? Does this mark the "end of stream"? This is currently how the code operates internally.
• Is the last flags not supposed to be added at all? Currently this is what is returned from the compressor.

I have a somewhat cryptic message in the Xpress compressor code which may be relevant:

// Finish shifting over flags and set all unused bytes to 1
// Note: the shifting math does not effect flags at all when flag_count == 0, resulting in a copy of the previous flags so the proper value must be set manually
// RTL produces improper output in this case as well, so the decompressor still must tolerate bad flags at the very end

[coderforlife]

As a side note, the Rtl version always requires an additional 24 bytes...

[nemequ]

FWIW I don't consider this urgent. I'm just using xpress_max_compressed_size() + 4 and everything works (though I have no idea about interop with the MS API).

By "the Rtl version", you mean MS's RtlCompressBuffer API? That will be a separate plugin, so that's not a problem.

[coderforlife]

Yes, by Rtl I mean the Rtl____ functions in ntdll.dll. They are may basis for comparison. There are plenty of other quirks with those functions as well. For example when decompressing Xpress Huffman data, the given output buffer must be exactly the right size (not larger or smaller) or it will fail. Both Xpress and Xpress Huffman Rtl compressors have the +24 byte requirement (meaning whatever the actual compressed output size is, they need 24 bytes more than that to work).

[coderforlife]

I have determined that those last 4 bytes (all 0xFF) are supposed to be added to the output (according to the RTL functions, but not so in the documentation which has plenty of errors in it), so the internal was right but the return values are wrong. I will fix both xpress_max_compressed_size() and xpress_compress().

[coderforlife]

This should now be fixed. Please check.