Extracting super big file

[a-v-ebrahimi]

I have a big zip file, 70mb, extracts to about 350mb. On low-end devices, app crashes because of low memory. How can I extract by chunk, not reading the whole file in memory?

[Nomy1]

Same. My file is 55mb and crashes from out of memory exception.

    zipFile = await zipFile.writeAsBytes(data, flush: true);
    final zipData = await zipFile.readAsBytes();

    // unzip file.
    return ZipDecoder().decodeBytes(
      zipData,
      verify: true,
      password: password,
    );

UPDATE: I removed verify and I can successfully decode it now.

[VERBOSE-2:ui_dart_state.cc(157)] Unhandled Exception: Out of Memory
#0      new Uint8List (dart:typed_data-patch/typed_data_patch.dart:2188:66)
#1      OutputStream._expandBuffer (package:archive/src/util/output_stream.dart:142:27)
#2      Inflate._decodeHuffman (package:archive/src/zlib/inflate.dart:49:47)
#3      Inflate._parseDynamicHuffmanBlock (package:archive/src/zlib/inflate.dart:238:5)
#4      Inflate._parseBlock (package:archive/src/zlib/inflate.dart:120:9)
#5      Inflate._inflate (package:archive/src/zlib/inflate.dart:90:19)
#6      new Inflate.buffer (package:archive/src/zlib/inflate.dart:19:5)
#7      ZipFile.content (package:archive/src/zip/zip_file.dart:94:28)
#8      ZipDecoder.decodeBuffer (package:archive/src/zip_decoder.dart:32:39)
#9      ZipDecoder.decodeBytes (package:archive/src/zip_decoder.dart:15:12)
#10     ZipDownloadApi._createArchive (package:qr_answers/services/api/filedownload/zip_download_api.dart:87:25)
#11     _rootRunUnary (dart:async/zone.dart:1134:38)
#12     _CustomZone.runUnary (dart:async/zone.dart:1031:19)
#13     _FutureListener.handleValue (dart:async/future_impl.dart:140:18)
#14     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:682:45)
#15     Future._propagateToListeners (dart:async/future_impl.dart:711:32)
#16     Future._completeWithValue (dart:async/future_impl.dart:526:5)
#17     Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:556:7)
#18     _rootRun (dart:async/zone.dart:1126:13)
#19     _CustomZone.run (dart:async/zone.dart:1023:19)
#20     _CustomZone.runGuarded (dart:async/zone.dart:925:7)
#21     _CustomZone.bindCallbackGuarded.<anonymous closure> (dart:async/zone.dart:965:23)
#22     _startMicrotaskLoop (dart:async/schedule_microtask.dart:43:21)```

[claudiofelber]

I had the same use case and therefore could not use the otherwise very good archive library. That's why I wrote async_zip. It allows reading a Zip file entry by entry from disk without it having to be completely loaded to memory. In addition read and write access can be asynchronous (there is also a slightly faster synchronous version available). Maybe it can solve your problem.

[albemala]

Confirm @Nomy1 update:

When passing verify: true, decompressing large files (>1GB for me) causes an Out of Memory exception.

However, if I remove the verify: true parameter, the app doesn't crash any more and the file gets extracted correctly.