[Debug] Exhausted heap space

[CaptainDario]

Currently, I am unzipping a moderately large file in my application (approx. 40mb). When doing so in release mode it works fine on all platforms, however, in debug mode, the unzipping fails on Android and ios. iOS shows this message (the one from android is similar)

Exhausted heap space, trying to allocate 1048608 bytes.

As there are is enough memory to unzip the file in release mode, I would like to ask how one can use this package so it also work in debug mode. I am trying to unzip this file using this code

await extractFileToDisk(
    "${destination.path}.zip",
    destination.parent.path
  );

[brendan-duncan]

If it's possible, maybe you could try the 4.0 branch I've been working on. You'd have to change your pubspec to the git path:

dependencies:
  archive:
    git:
      url: https://github.com/brendan-duncan/archive
      ref: '4.0'

I did a bunch of work on memory and file io performance in that branch, it would be great to know if it's enough to get past the iOS memory issues. It should also be considerably faster to extract that zip too. I tested the previous version and it took over a minute, and on the 4.0 branch it took a couple seconds.

[CaptainDario]

Thank you for your reply! I will try this today in the evening, but this does not only happen on iOS but also on Android.

[brendan-duncan]

Mobile is much more restrictive with memory. The 4.0 branch is now in the main branch, it will be the next version update. So you can test without the git branch:

dependencies:
  archive:
    git:
      url: https://github.com/brendan-duncan/archive

[CaptainDario]

Is there a migration guide available? Hitting this

The method 'decodeBuffer' isn't defined for the type 'ZipDecoder'.
Try correcting the name to the name of an existing method, or defining a method named 'decodeBuffer'.

[CaptainDario]

I assume it should be decodeStream but I am not sure.

[brendan-duncan]

Yeah, decodeStream. There's a migration guide link in the README.

[CaptainDario]

I tried the new version in debug mode on Android and it is significantly slower, even so much that I cannot use this version.

v3.6.1

I/flutter (18876): assets/dict/dictionary.zip
I/flutter (18876): 0:00:01.509784
I/flutter (18876): assets/dict/examples.zip
I/flutter (18876): 0:00:01.104178
I/flutter (18876): assets/dict/krad.zip
I/flutter (18876): 0:00:00.250422
I/flutter (18876): assets/dict/radk.zip
I/flutter (18876): 0:00:00.170664
I/flutter (18876): assets/ipadic.zip
I/flutter (18876): 0:00:00.384508

master

I/flutter (14027): assets/dict/dictionary.zip
I/flutter (14027): 0:02:35.098774
I/flutter (14027): assets/dict/examples.zip
I/flutter (14027): 0:01:21.271891
I/flutter (14027): assets/dict/krad.zip
I/flutter (14027): 0:00:00.988176
I/flutter (14027): assets/dict/radk.zip
I/flutter (14027): 0:00:00.564761
I/flutter (14027): assets/ipadic.zip
I/flutter (14027): 0:00:26.827676

You can find the used files here.

[CaptainDario]

But on a positive note, the unpacking of the initially mentioned zip does now work on android in debug mode. Thanks!

[CaptainDario]

I could add decodeBuffer back with a deprecated label if you think that would be easier.

No, I think it is fine. I just did not know where to find the migration guide

[brendan-duncan]

Well shoot, back to the drawing board then. Sometimes Dart drives me a bit nuts. Must be some extra file io, probably file position seeking, causing the slowdown. Strange the tests I did were faster. Thanks for the report, I'll get it sorted out soon.

[brendan-duncan]

Looks like I'll need to rework the cache for the zlib decompression a bit. The file seeking it needs to do is going outside the bounds of the in-memory cache, and causing file io thrashing. I had previously improved it for smaller compressed blocks, like in that Audios zip, but the other zips in your collection have much larger compressed blocks. Hopefully I can get that fixed up soon.

[brendan-duncan]

Decoding a zip on dart:io platforms should be significantly faster with 4.0 now.

It will use native executed zlib for decompressing zip data on platforms that support it now. The dictionary.zip file was taking >1minute before, takes <1second now.

If you could run your tests on the update in git, that would be appreciated.

[timautin]

Same problem here using latest 3.x or 4.0. I'm trying to unzip a 2.5 GB archive containing a 1 GB file, it fails on iOS (working on Android). Here's the code:

final input = InputFileStream("/path/to/my-archive.zip");
final archive = ZipDecoder().decodeStream(input);

for (final file in archive) {
  final filename = file.name;
  if (file.isFile) {
    final output = OutputFileStream('/path/to/$filename');
    output.writeStream(file.getContent()!);
    await file.close();
  } else {
    await Directory('${datasetsFolder.path}/$filename').create(recursive: true);
  }
}

And the log:

Exhausted heap space, trying to allocate 1044090912 bytes. [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Out of Memory

0 new Uint8List (dart:typed_data-patch/typed_data_patch.dart:2236:3)

1 new OutputMemoryStream (package:archive/src/util/output_memory_stream.dart:17:19)

2 new Inflate.stream (package:archive/src/codecs/zlib/inflate.dart:37:29)

3 ZipFile.getStream (package:archive/src/codecs/zip/zip_file.dart:232:21)

4 ArchiveFile.decompress (package:archive/src/archive/archive_file.dart:208:40)

5 ArchiveFile.getContent (package:archive/src/archive/archive_file.dart:139:7)

6 ApplicationState.test (package:atlas/application.dart:77:33)

7 _FutureListener.handleValue (dart:async/future_impl.dart:162:18)

Are you planning to support this use case or do I need to find a workaround?

[CaptainDario]

@brendan-duncan sorry for the long wait and thank you for further looking into this!

I checked the current version by using

archive:
    git:
      url: https://github.com/brendan-duncan/archive

The speed is great again (it seems to be even faster, at least for my use case)

I/flutter (13041): 0:00:01.689626
I/flutter (13041): assets/dict/examples.zip
I/flutter (13041): 0:00:01.288181
I/flutter (13041): assets/dict/krad.zip
I/flutter (13041): 0:00:00.173609
I/flutter (13041): assets/dict/radk.zip
I/flutter (13041): 0:00:00.187659
I/flutter (13041): assets/ipadic.zip
I/flutter (13041): 0:00:00.420636

Also on my phone, there is no out of memory error in debug mode anymore, thank you so much!