DeflateStream can't handle input that exceeds the maximum number of non-compressible blocks

[theseion]

Migrated from https://pharo.fogbugz.com/f/cases/20973/DeflateStream-can-t-handle-input-that-exceeds-the-maximum-number-of-non-compressible-blocks //-----

So, I seem to have identified the problem. According to the RFC on Deflate the maximum number of non-compressible blocks is limited to (2^16) - 1:

"A compressed data set consists of a series of blocks, corresponding to successive blocks of input data. The block sizes are arbitrary, except that non-compressible blocks are limited to 65,535 bytes."

Stephan's case exceeds this limit, which leads to an error when validating the match in DeflateStream>>validateMatchAt:from:to:. I assume that the compiler optimization removes the trap for the validation failure in this case (note that I simply used the Smalltalk implementation to trace the problem by replacing the primitive send with a super send).

With Fuel loaded (Pharo 7, 64-bit) the following produces a reproducible validation error, which is misleading, the problem is actually that the number of blocks was exceeded (I think that's because of the 16 bit bit mask that is being used in the implementation):

(FLGZipStrategy newWithTarget: FLByteArrayStreamStrategy new) writeStreamDo: [ :stream |
FLSerializer newDefault
serialize: ((1 to: (2 raisedTo: 16) - 82) collect: [ :i | 1 ]) asByteArray
on: stream ]

I'm not sure what should be done in this case but it appears to me that all remaining data could simply be appended to the last block, at least in the implementation I found in GCC:

   /* Stored blocks are limited to 0xffff bytes, pending_buf is limited
    * to pending_buf_size, and each stored block has a 5 byte header:
   */
    ulg max_block_size = 0xffff;
    ulg max_start;

    if (max_block_size > s->pending_buf_size - 5) {
        max_block_size = s->pending_buf_size - 5;
    }

We either need to handle exceeding the maximum non-compressible block size gracefully (clear error message) or, what I'd prefer, make this problem go away. If appending to the last block is not possible, maybe we should switch to Deflate64.

I've also opened an issue for the VM about fixing the crashing problem when the primitive fails (https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/193).

[stale[bot]]

To limit bug bankruptcy (see https://www.joelonsoftware.com/2012/07/09/software-inventory/) this issue has been automatically marked as stale because it has not had any activity in 8 months. It will be closed in 1 month if no further activity occurs. If this issue remains important to you, please comment to reactivate the issue. Thank you for your contributions.

Joel on Software

Software Inventory

Imagine, for a moment, that you came upon a bread factory for the first time. At first it just looks like a jumble of incomprehensible machinery with a few people buzzing around. As your eyes adjus…

[theseion]

Still relevant

[stale[bot]]

To limit bug bankruptcy (see https://www.joelonsoftware.com/2012/07/09/software-inventory/) this issue has been automatically marked as stale because it has not had any activity in 6 months. If no further activity occurs, it might be closed. If this issue remains important to you, please comment to reactivate the issue. Thank you for your contributions.

Joel on Software

Software Inventory

Imagine, for a moment, that you came upon a bread factory for the first time. At first it just looks like a jumble of incomprehensible machinery with a few people buzzing around. As your eyes adjus…

[theseion]

Still relevant