Flushing not working as expected

[nemequ]

I'm trying to add support for flushing to the ms-compress squash plugin, but it isn't working as I would expect. Perhaps I'm missing something, but here is a quick test case for how I would expect it to work:

#include <stdbool.h>
#include <assert.h>
#include <stdio.h>

#include "include/mscomp.h"

#define LOREM_IPSUM                                                     \
  "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vulputate " \
  "lectus nisl, vitae ultricies justo dictum nec. Vestibulum ante ipsum " \
  "primis in faucibus orci luctus et ultrices posuere cubilia Curae; "  \
  "Suspendisse suscipit quam a lectus adipiscing, sed tempor purus "    \
  "cursus. Vivamus id nulla eget elit eleifend molestie. Integer "      \
  "sollicitudin lorem enim, eu eleifend orci facilisis sed. Pellentesque " \
  "sodales luctus enim vel viverra. Cras interdum vel nisl in "         \
  "facilisis. Curabitur sollicitudin tortor vel congue "                \
  "auctor. Suspendisse egestas orci vitae neque placerat blandit.\n"    \
  "\n"                                                                  \
  "Aenean sed nisl ultricies, vulputate lorem a, suscipit nulla. Donec " \
  "egestas volutpat neque a eleifend. Nullam porta semper "             \
  "nunc. Pellentesque adipiscing molestie magna, quis pulvinar metus "  \
  "gravida sit amet. Vestibulum mollis et sapien eu posuere. Quisque "  \
  "tristique dignissim ante et aliquet. Phasellus vulputate condimentum " \
  "nulla in vulputate.\n"                                               \
  "\n"                                                                  \
  "Nullam volutpat tellus at nisi auctor, vitae mattis nibh viverra. Nunc " \
  "vitae lectus tristique, ultrices nibh quis, lobortis elit. Curabitur " \
  "at vestibulum nisi, nec facilisis ante. Nulla pharetra blandit lacus, " \
  "at sodales nulla placerat eget. Nulla congue varius tortor, sit amet " \
  "tempor est mattis nec. Praesent vitae tristique ipsum, rhoncus "     \
  "tristique lorem. Sed et erat tristique ligula accumsan fringilla eu in " \
  "urna. Donec dapibus hendrerit neque nec venenatis. In euismod sapien " \
  "ipsum, auctor consectetur mi dapibus hendrerit.\n"                   \
  "\n"                                                                  \
  "Phasellus sagittis rutrum velit, in sodales nibh imperdiet a. Integer " \
  "vitae arcu blandit nibh laoreet scelerisque eu sit amet eros. Aenean " \
  "odio felis, aliquam in eros at, ornare luctus magna. In semper "     \
  "tincidunt nunc, sollicitudin gravida nunc laoreet eu. Cras eu tempor " \
  "sapien, ut dignissim elit. Proin eleifend arcu tempus, semper erat et, " \
  "accumsan erat. Praesent vulputate diam mi, eget mollis leo "         \
  "pellentesque eget. Aliquam eu tortor posuere, posuere velit sed, "   \
  "suscipit eros. Nam eu leo vitae mauris condimentum lobortis non quis " \
  "mauris. Nulla venenatis fringilla urna nec venenatis. Nam eget velit " \
  "nulla. Proin ut malesuada felis. Suspendisse vitae nunc neque. Donec " \
  "faucibus tempor lacinia. Vivamus ac vulputate sapien, eget lacinia " \
  "nisl.\n"                                                             \
  "\n"                                                                  \
  "Curabitur eu dolor molestie, ullamcorper lorem quis, egestas "       \
  "urna. Suspendisse in arcu sed justo blandit condimentum. Ut auctor, " \
  "sem quis condimentum mattis, est purus pulvinar elit, quis viverra " \
  "nibh metus ac diam. Etiam aliquet est eu dui fermentum consequat. Cras " \
  "auctor diam eget bibendum sagittis. Aenean elementum purus sit amet " \
  "sem euismod, non varius felis dictum. Aliquam tempus pharetra ante a " \
  "sagittis. Curabitur ut urna felis. Etiam sed vulputate nisi. Praesent " \
  "at libero eleifend, sagittis quam a, varius sapien."
#define LOREM_IPSUM_LENGTH 2725

int main(int argc, char *argv[])
{
  mscomp_stream comp;
  mscomp_stream decomp;
  MSCompStatus status;
  unsigned char compressed[LOREM_IPSUM_LENGTH * 4];
  unsigned char decompressed[LOREM_IPSUM_LENGTH];

  status = ms_deflate_init (MSCOMP_LZNT1, &comp);
  assert (status == MSCOMP_OK);
  status = ms_inflate_init (MSCOMP_LZNT1, &decomp);
  assert (status == MSCOMP_OK);

  comp.in = (const_bytes) LOREM_IPSUM;
  comp.in_avail = LOREM_IPSUM_LENGTH;
  comp.out = compressed;
  comp.out_avail = sizeof(compressed);

  status = ms_deflate(&comp, MSCOMP_NO_FLUSH);
  assert (status == MSCOMP_OK);

  status = ms_deflate(&comp, MSCOMP_FLUSH);
  assert (status == MSCOMP_OK);

  decomp.in = compressed;
  decomp.in_avail = comp.out_total;
  decomp.out = decompressed;
  decomp.out_avail = sizeof(decompressed);

  status = ms_inflate(&decomp);
  assert (status == MSCOMP_OK);

  assert (comp.out_total == LOREM_IPSUM_LENGTH);

  ms_deflate_end(&comp);
  ms_inflate_end (&decomp);

  return 0;
}

The really surprising part is that after calling ms_deflate with MSCOMP_FLUSH no data is written to the output.

[coderforlife]

At least with the latest in master, this mostly works. I have done some work on the flushing system of LZNT1, so check it out.

After the first ms_deflate, I have: in_avail 0x00000000 in_total 0x00000aa5 out_avail 0x00002a94 out_total 0x00000000

After the second ms_deflate, I have: in_avail 0x00000000 in_total 0x00000aa5 out_avail 0x000024e3 out_total 0x000005b1

Shows that no data is written during the first ms_deflate (understandable since LZNT1 wiats for 4096 bytes to compress) and when calling with MSCOMP_FLUSH it outputs 0x05B1 bytes.

There are some problems with your code in the newest master:

• ms_inflate actually returns MSCOMP_POSSIBLE_STREAM_END here and not MSCOMP_OK because it realizes that the stream is probably ended
• Your last assert should probably be decomp and not comp
• You probably meant to use MSCOMP_FINISH instead of MSCOMP_FLUSH - while for LZNT1 they are basically identical (although ms_inflate_end will return an error saying not finished), but more importantly other formats are likely not to support FLUSH as well as FINISH. Additionally, FINISH won't return MSCOMP_OK but instead MSCOMP_STREAM_END.

I have gone through and documented mscomp.h much better now. It should explain a lot more. But in general, I have made several fixes to streaming stuff recently. Xpress still does not stream-compress, but I am holding off on finishing that until I get the speed of the general compression up.

[nemequ]

ms_inflate actually returns MSCOMP_POSSIBLE_STREAM_END here and not MSCOMP_OK because it realizes that the stream is probably ended

Ah, don't know how I missed that >:(

Your last assert should probably be decomp and not comp

Yep. At that point I was just throwing stuff in with no way to test.

You probably meant to use MSCOMP_FINISH instead of MSCOMP_FLUSH - while for LZNT1 they are basically identical (although ms_inflate_end will return an error saying not finished), but more importantly other formats are likely not to support FLUSH as well as FINISH. Additionally, FINISH won't return MSCOMP_OK but instead MSCOMP_STREAM_END.

In this case I just wanted an example of flush working correctly, but yes, in real code I'll make the distinction between flushing and finishing.

I have gone through and documented mscomp.h much better now. It should explain a lot more. But in general, I have made several fixes to streaming stuff recently. Xpress still does not stream-compress, but I am holding off on finishing that until I get the speed of the general compression up.

Oh, wow, that's a lot better than last time I looked at it. Sorry, I really should have checked your recent changes before bothering you with this.