Cannot decompress to a buffer exactly the right size

[nemequ]

#define LOREM_IPSUM                                                     \
  "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vulputate " \
  "lectus nisl, vitae ultricies justo dictum nec. Vestibulum ante ipsum " \
  "primis in faucibus orci luctus et ultrices posuere cubilia Curae; "  \
  "Suspendisse suscipit quam a lectus adipiscing, sed tempor purus "    \
  "cursus. Vivamus id nulla eget elit eleifend molestie. Integer "      \
  "sollicitudin lorem enim, eu eleifend orci facilisis sed. Pellentesque " \
  "sodales luctus enim vel viverra. Cras interdum vel nisl in "         \
  "facilisis. Curabitur sollicitudin tortor vel congue "                \
  "auctor. Suspendisse egestas orci vitae neque placerat blandit.\n"    \
  "\n"                                                                  \
  "Aenean sed nisl ultricies, vulputate lorem a, suscipit nulla. Donec " \
  "egestas volutpat neque a eleifend. Nullam porta semper "             \
  "nunc. Pellentesque adipiscing molestie magna, quis pulvinar metus "  \
  "gravida sit amet. Vestibulum mollis et sapien eu posuere. Quisque "  \
  "tristique dignissim ante et aliquet. Phasellus vulputate condimentum " \
  "nulla in vulputate.\n"                                               \
  "\n"                                                                  \
  "Nullam volutpat tellus at nisi auctor, vitae mattis nibh viverra. Nunc " \
  "vitae lectus tristique, ultrices nibh quis, lobortis elit. Curabitur " \
  "at vestibulum nisi, nec facilisis ante. Nulla pharetra blandit lacus, " \
  "at sodales nulla placerat eget. Nulla congue varius tortor, sit amet " \
  "tempor est mattis nec. Praesent vitae tristique ipsum, rhoncus "     \
  "tristique lorem. Sed et erat tristique ligula accumsan fringilla eu in " \
  "urna. Donec dapibus hendrerit neque nec venenatis. In euismod sapien " \
  "ipsum, auctor consectetur mi dapibus hendrerit.\n"                   \
  "\n"                                                                  \
  "Phasellus sagittis rutrum velit, in sodales nibh imperdiet a. Integer " \
  "vitae arcu blandit nibh laoreet scelerisque eu sit amet eros. Aenean " \
  "odio felis, aliquam in eros at, ornare luctus magna. In semper "     \
  "tincidunt nunc, sollicitudin gravida nunc laoreet eu. Cras eu tempor " \
  "sapien, ut dignissim elit. Proin eleifend arcu tempus, semper erat et, " \
  "accumsan erat. Praesent vulputate diam mi, eget mollis leo "         \
  "pellentesque eget. Aliquam eu tortor posuere, posuere velit sed, "   \
  "suscipit eros. Nam eu leo vitae mauris condimentum lobortis non quis " \
  "mauris. Nulla venenatis fringilla urna nec venenatis. Nam eget velit " \
  "nulla. Proin ut malesuada felis. Suspendisse vitae nunc neque. Donec " \
  "faucibus tempor lacinia. Vivamus ac vulputate sapien, eget lacinia " \
  "nisl.\n"                                                             \
  "\n"                                                                  \
  "Curabitur eu dolor molestie, ullamcorper lorem quis, egestas "       \
  "urna. Suspendisse in arcu sed justo blandit condimentum. Ut auctor, " \
  "sem quis condimentum mattis, est purus pulvinar elit, quis viverra " \
  "nibh metus ac diam. Etiam aliquet est eu dui fermentum consequat. Cras " \
  "auctor diam eget bibendum sagittis. Aenean elementum purus sit amet " \
  "sem euismod, non varius felis dictum. Aliquam tempus pharetra ante a " \
  "sagittis. Curabitur ut urna felis. Etiam sed vulputate nisi. Praesent " \
  "at libero eleifend, sagittis quam a, varius sapien."
#define LOREM_IPSUM_LENGTH 2725

#include <assert.h>
#include <stddef.h>

#include "src/density_api.h"

int main(int argc, char *argv[]) {
  uint8_t compressed[4096];
  size_t compressed_length = sizeof(compressed);
  uint8_t decompressed[LOREM_IPSUM_LENGTH];
  size_t decompressed_length = sizeof(decompressed);
  density_buffer_processing_result res;

  res = density_buffer_compress (LOREM_IPSUM, LOREM_IPSUM_LENGTH,
                                 compressed, compressed_length,
                                 DENSITY_COMPRESSION_MODE_CHAMELEON_ALGORITHM, DENSITY_BLOCK_TYPE_DEFAULT,
                                 NULL, NULL);

  assert (res.state == DENSITY_BUFFER_STATE_OK);

  compressed_length = res.bytesWritten;

  res = density_buffer_decompress (compressed, compressed_length,
                                   decompressed, decompressed_length,
                                   NULL, NULL);

  assert (res.state == DENSITY_BUFFER_STATE_OK);
  assert (res.bytesWritten == LOREM_IPSUM_LENGTH);
  assert (memcmp (decompressed, LOREM_IPSUM, LOREM_IPSUM_LENGTH) == 0);

  return 0;
}

[g1mv]

Yes it's a known behavior, decompression actually needs at least DENSITY_CHAMELEON_DECODE_MINIMUM_OUTPUT_LOOKAHEAD or DENSITY_MANDALA_DECODE_MINIMUM_OUTPUT_LOOKAHEAD bytes to continue processing. I don't know if this can be filed as a problem or not, because you just need a few hundred extra bytes or trick the decompressor into thinking it has the bytes available for everything to work properly. In your case, if you use LOREM_IPSUM_LENGTH + DENSITY_CHAMELEON_DECODE_MINIMUM_OUTPUT_LOOKAHEAD as the decompressed buffer length, everything will work, but I agree it's not ideal.

[nemequ]

For a simple test case like this that would work, but there are plenty of cases where that's not feasible and you would really have no choice but to decompress to a temporary buffer then memcpy over to the real buffer. I would actually guess that most people who would be interested in the buffer to buffer API would fall into that case.

Maybe you could provide _fast and _safe versions of the API (kind of like LZ4, as well as some others)? Or tweak the implementation so that the last block decompresses to a temporary buffer if there is not enough space available to decompress it directly to the supplied buffer?

[g1mv]

Implemented in 9e28aa59d586bc8fa7c80e85aff9c26fd6446bf5