Can I decompress in segments when I run out of memory?

[baloyou]

Can I decompress in segments when I run out of memory? (Does the memory have to be larger than the full data)

---

I use this library (1.3) (esp8266+arduino) on my microcontroller to decompress the requested API data (the raw data is gzip compressed)

I used the following code, and when the raw data is small, everything works fine. However, if the original data exceeds the settings of inputbuffer and outputbuffer, decompression will fail.

int result = inflate(&zlibStream, Z_SYNC_FLUSH);

This code returns Z_MEM_ERROR

      uint8_t inputBuffer[512]; 
      uint8_t outputBuffer[512]; 

I can't expand the values of inputbuffer and outputbuffer because ESP8266 has limited memory space (can only be set up to 512)

How can I solve this problem?

I'm sorry, but I don't know much about gzip's data format.

full code:

  if (client.connect(serverName, serverPort)) {
    String request = "GET " + serverPath + " HTTP/1.1\r\n" +
                     "Host: " + serverName + "\r\n" +
                     "Connection: close\r\n\r\n";

    client.print(request);

    while (client.connected()) {
      String line = client.readStringUntil('\n');
      Serial.println(line);
      if (line == "\r") {
        break;
      }
    }

    z_stream zlibStream;
    zlibStream.zalloc = Z_NULL;
    zlibStream.zfree = Z_NULL;
    zlibStream.opaque = Z_NULL;
    zlibStream.avail_in = 0;
    zlibStream.next_in = Z_NULL;

    if (inflateInit2(&zlibStream, 16 + MAX_WBITS) == Z_OK) {
      String uncompressedData;
      uint8_t inputBuffer[512]; // Adjust the buffer size as needed
      uint8_t outputBuffer[512]; // Adjust the buffer size as needed

      while (client.available()) {
        int bytesRead = client.readBytes(inputBuffer, sizeof(inputBuffer));
        zlibStream.avail_in = bytesRead;
        zlibStream.next_in = inputBuffer;

        do {
          zlibStream.avail_out = sizeof(outputBuffer);
          zlibStream.next_out = outputBuffer;

          int result = inflate(&zlibStream, Z_SYNC_FLUSH);
          if (result == Z_STREAM_END) {
            int decompressedLength = sizeof(outputBuffer) - zlibStream.avail_out;
            uncompressedData += String((char*)outputBuffer).substring(0, decompressedLength);
            break;  // Exit the loop when decompression is complete
          } else if (result == Z_OK) {
            int decompressedLength = sizeof(outputBuffer) - zlibStream.avail_out;
            uncompressedData += String((char*)outputBuffer).substring(0, decompressedLength);
          } else {
            Serial.print(result);
            Serial.println("Decompression error");

            Serial.print("Decompression error: ");
            Serial.println(ESP.getFreeHeap());
            break;
          }
        } while (zlibStream.avail_out == 0);
      }

      Serial.println(uncompressedData);
      Serial.println(uncompressedData.length());
      inflateEnd(&zlibStream);

[Neustradamus]

@madler: Can you look this ticket?

[Neustradamus]

@madler: Can you look this ticket?

[madler]

Can I decompress in segments when I run out of memory? (Does the memory have to be larger than the full data)

Yes, but if you decompress in chunks, then you need 32K of memory to save the sliding window for decompression. inflate() will only try to allocate that 32K if it cannot complete the decompression in a single call.