Continous loop after deflate() error

[michaelrsweet]

Version: 1.8-current Original reporter:

htmldoc doesn't check the return code from deflate(). If deflate() fails, this can lead to a continuous loop in flate_write().

This problem is difficult to reproduce, however I have a particular document which fails in this manner when particular ulimit settings are in effect. I originally saw the problem in 1.8.23, and I get the same result with 1.8.25.

I've attached a diff against 1.8.25 with my work-around to the problem.

[michaelrsweet]

Original reporter: Michael Sweet

No one has reported this problem in the last 8 years we've been using this code... :)

If we do a 1.8.26 release, we'll make sure something similar is added to the code (calling abort() is not necessarily the right thing to do), but regardless it will definitely be addressed in 1.9.

[michaelrsweet]

Original reporter: Michael Sweet

Fixed in Subversion repository.

[michaelrsweet]

"ps-pdf.patch":

--- ps-pdf.cxx.orig 2005-11-03 11:27:45.000000000 -0500
+++ ps-pdf.cxx  2005-11-30 09:58:15.000000000 -0500
@@ -12562,6 +12562,8 @@ flate_open_stream(FILE *out)        /* I - Out
 static void
 flate_close_stream(FILE *out)      /* I - Output file */
 {
+  int rc;
+
   if (!Compression)
   {
 #ifdef HTMLDOC_ASCII85
@@ -12572,8 +12574,16 @@ flate_close_stream(FILE *out)      /* I - Ou
     return;
   }

-  while (deflate(&compressor, Z_FINISH) != Z_STREAM_END)
+  while ((rc = deflate(&compressor, Z_FINISH)) != Z_STREAM_END)
   {
+    if (rc < Z_OK && rc != Z_BUF_ERROR)
+    {
+          progress_error(HD_ERROR_OUT_OF_MEMORY,
+                         "deflate() failed (%d)", rc);
+          abort();
+    }
+
+
     if (PSLevel)
 #ifdef HTMLDOC_ASCII85
       ps_ascii85(out, comp_buffer,
@@ -12673,6 +12683,7 @@ flate_write(FILE  *out,         /* I - Output 
             int   length,      /* I - Number of bytes to write */
        int   flush)        /* I - Flush when writing data? */
 {
+  int rc;
   if (compressor_active)
   {
     compressor.next_in  = buf;
@@ -12704,7 +12715,12 @@ flate_write(FILE  *out,            /* I - Output 
    compressor.avail_out = sizeof(comp_buffer);
       }

-      deflate(&compressor, flush ? Z_FULL_FLUSH : Z_NO_FLUSH);
+      if ((rc = deflate(&compressor, flush ? Z_FULL_FLUSH : Z_NO_FLUSH)) < Z_OK && rc != Z_BUF_ERROR)
+      {
+          progress_error(HD_ERROR_OUT_OF_MEMORY,
+                         "deflate() failed (%d)", rc);
+          abort();
+      }
       flush = 0;
     }
   }