How to extract in async mode?

[paulocoutinhox]

Hi,

How to extract in async mode?

When im extracting data from zip it is freezing my UI.

Thanks.

[brendan-duncan]

For now, you'll need to do the extract in an Isolate thread.

[leecommamichael]

You might wonder why this package doesn't just expose an async function. It's not impossible, but someone will need to have code that, when compiled for native, will use Isolates, and when compiled for the browser, will use ServiceWorkers.

Because the way to create threaded-work is different between the browser and native, it makes it difficult for packages to implement in a platform-independent way.

[brendan-duncan]

@leecommamichael Yes, definitely. Dart used to be primarily a web language. It pivoted to mobile when Flutter came out. Async and threading is one of the things that is very different between mobile and web, and supporting both is not simple at all. In my imaginary "free time" I would split the package up and have a Flutter specific implementation with all of the niceties of async, native implemented backend functions, streaming, etc.

[claudiofelber]

Because I needed asynchronous reading and writing capabilities of Zip files with low memory consumption I wrote async_zip. Maybe it works for your use case, too.

[paulocoutinhox]

This is possible with recent versions?

[sidetraxaudio]

For now, you'll need to do the extract in an Isolate thread.

Brendan, the core compression function seems to be efficiently multi-threaded, which is typically good, however on a large compression task it completely maxes out all cores on the cpu at 100% (windows desktop). Spawning on an isolate allows UI functions on the main thread to slip through as expected, but as a whole, system responsiveness grinds to a halt. In my case all 8 cores. Any way of passing max threads to the encoder?

[brendan-duncan]

@sidetraxaudio I'm not sure what you mean by efficiently multthreaded. Dart is not multi-threaded beyond isolates, so running in an isolate would give you one thread for compression.

[sidetraxaudio]

@sidetraxaudio I'm not sure what you mean by efficiently multthreaded. Dart is not multi-threaded beyond isolates, so running in an isolate would give you one thread for compression.

@brendan-duncan It could if it passes stuff off to the OS or a non-dart package. I've just started using and I'm not sure how it works under the hood. If its plain dart then obviously you're correct, however for some reason on a 2GB compression I'm getting a full max out on one core (expected) and near max out on 8 cores only while running the compression. I also get intermittent mouse hangs across the whole OS. Using winrar on the same files utilizes even more CPU with no mouse or OS anomalies. I'm at a loss as to why.

[sidetraxaudio]

@sidetraxaudio I'm not sure what you mean by efficiently multthreaded. Dart is not multi-threaded beyond isolates, so running in an isolate would give you one thread for compression.

Forget last comment about OS hangs. Its memory swapping to the pagefile due to memory utilisation, not any kind of blocking.

[brendan-duncan]

There's the pachage:archive/archive_io.dart variant that has file stream classes to avoid having to load the entire archive file into memory.

import 'package:archive/archive_io.dart';
void main() {
  final stream = InputFileStream(p.join(testDirPath, 'res/test.zip'));
  final zip = ZipDecoder().decodeBuffer(stream);
  //...
}

[brendan-duncan]

This whole library is overdue for a refresh/rewrite, which I plan on doing soon.

[sidetraxaudio]

There's the pachage:archive/archive_io.dart variant that has file stream classes to avoid having to load the entire archive file into memory.

import 'package:archive/archive_io.dart';
void main() {
  final stream = InputFileStream(p.join(testDirPath, 'res/test.zip'));
  final zip = ZipDecoder().decodeBuffer(stream);
  //...
}

Great good idea thanks. I'll work out the equivalent functions for the encoder and rewrite/check for memory. I'm using archive_io already but calling ZipFileEncoder.create and then .addfile. It looks like a great package. A rewrite with a few comments on the various methods would be highly appreciated by the community.