search

gildas-lormeau / zip.js

JavaScript library to zip and unzip files supporting multi-core compression, compression streams, zip64, split files and encryption.
https://gildas-lormeau.github.io/zip.js
BSD 3-Clause "New" or "Revised" License
3.4k stars 510 forks source link

Corruption when zipping fast #504

Closed DebianArch64 closed 5 months ago

DebianArch64 commented 5 months ago

Probably something on my end, but for some reason Zip files are more corrupt when having no delay between adding files. So, there are two issues but hoping to have them both fixed on my side

The Output Output without delay:

Archive:  /Users/debianarch/Downloads/install (11).ipa
  inflating: Payload/Taurine.app/essential_0-4_iphoneos-arm.deb  
  inflating: Payload/Taurine.app/bootstrap.tar.gz  
  error:  invalid compressed data to inflate
file #3:  bad zipfile offset (local header sig):  12594496
file #4:  bad zipfile offset (local header sig):  12603319
file #5:  bad zipfile offset (local header sig):  12617048
file #6:  bad zipfile offset (local header sig):  15534336
file #7:  bad zipfile offset (local header sig):  15535204
file #8:  bad zipfile offset (local header sig):  15535920
file #9:  bad zipfile offset (local header sig):  15536177
file #10:  bad zipfile offset (local header sig):  15776964
file #11:  bad zipfile offset (local header sig):  15777490
file #12:  bad zipfile offset (local header sig):  15779564
file #13:  bad zipfile offset (local header sig):  15780306
file #14:  bad zipfile offset (local header sig):  15797892
file #15:  bad zipfile offset (local header sig):  15799206
file #16:  bad zipfile offset (local header sig):  15799489
file #17:  bad zipfile offset (local header sig):  28942874
file #18:  bad zipfile offset (local header sig):  28961940
file #19:  bad zipfile offset (local header sig):  28968084
  inflating: Payload/Taurine.app/signcert.p12  
  inflating: Payload/Taurine.app/Info.plist  
  inflating: Payload/Taurine.app/PkgInfo  
  inflating: Payload/Taurine.app/_CodeSignature/CodeResources  
  inflating: Payload/Taurine.app/Taurine

Output with delay:

Archive:  /Users/debianarch/Downloads/install (13).ipa
  inflating: Payload/Taurine.app/essential_0-4_iphoneos-arm.deb  
  inflating: Payload/Taurine.app/bootstrap.tar.gz  
  error:  invalid compressed data to inflate
  inflating: Payload/Taurine.app/uicache.gz  
  inflating: Payload/Taurine.app/AppIcon60x60@2x.png  
  inflating: Payload/Taurine.app/basebinaries.tar  
  error:  invalid compressed data to inflate
  inflating: Payload/Taurine.app/Base.lproj/LaunchScreen.storyboardc/01J-lp-oVM-view-Ze5-6b-2t3.nib  
  inflating: Payload/Taurine.app/Base.lproj/LaunchScreen.storyboardc/UIViewController-01J-lp-oVM.nib  
  inflating: Payload/Taurine.app/Base.lproj/LaunchScreen.storyboardc/Info.plist  
  inflating: Payload/Taurine.app/tar.gz  
  inflating: Payload/Taurine.app/BuildConfigs.xcconfig  
  inflating: Payload/Taurine.app/Main.storyboardc/DKI-Yp-loo-view-cPN-qX-cZW.nib  
  inflating: Payload/Taurine.app/Main.storyboardc/UIViewController-DKI-Yp-loo.nib  
  inflating: Payload/Taurine.app/Main.storyboardc/BYZ-38-t0r-view-8bC-Xf-vdC.nib  
  inflating: Payload/Taurine.app/Main.storyboardc/UINavigationController-veQ-yo-6fF.nib  
  inflating: Payload/Taurine.app/Main.storyboardc/Info.plist  
  inflating: Payload/Taurine.app/Assets.car  
  error:  invalid compressed data to inflate
  inflating: Payload/Taurine.app/AppIcon76x76@2x~ipad.png  
  inflating: Payload/Taurine.app/ctbypass.gz  
  inflating: Payload/Taurine.app/org.coolstar.sileo_2.3_iphoneos-arm.deb  
  inflating: Payload/Taurine.app/signcert.p12  
  inflating: Payload/Taurine.app/Info.plist  
  inflating: Payload/Taurine.app/PkgInfo  
  inflating: Payload/Taurine.app/_CodeSignature/CodeResources  
  inflating: Payload/Taurine.app/Taurine

The Code Pretty much, I have an emscripten WASM directory that I'm zipping up using zip.js. To be speedy I am uploading the chunks to a websocket as it's being zipped up:

  const writer =
    new zip.ZipWriterStream({
      dataDescriptor: true, // this is required for streams ??? https://github.com/gildas-lormeau/zip.js/issues/274#issuecomment-923032335
      extendedTimestamp: false,

      keepOrder: true,
    });

    -----
   writer.readable.pipeTo(
    new WritableStream({
      write: (chunk) => {
         // here I send the chunks to a websocket
      },
      close: () => {
        console.error("closing socket");
        socket.close();
      },
      abort: () => {
        console.error("aborting socket");
        socket.close();
      },
    }),
    { preventClose: true }
  );

  ----

  const test = new Blob([
        FS.readFile(path, { flags: "r", encoding: "binary" }) as Uint8Array,
      ]);
      // await delay(1000); // this is the delay i need to add for things to be less broken
      await writer.zipWriter.add(inferedPath, new zip.BlobReader(test));
gildas-lormeau commented 5 months ago

Did you implement anything to ensure that websocket messages are ordered or does your environment guarantee it? See https://stackoverflow.com/questions/11804721/can-websocket-messages-arrive-out-of-order

DebianArch64 commented 5 months ago

Did you implement anything to ensure that websocket messages are ordered or does your environment guarantee it? See https://stackoverflow.com/questions/11804721/can-websocket-messages-arrive-out-of-order

The websocket isn't an issue, it's receiving the same amount of bytes and in the right order (I printed out size of bytes received in my erlang backend and size of bytes written from JavaScript) so the zip I'm making using .add function is what's wonky, it's really a few of the exact same entries that aren't writing correctly (the failed to inflate message for the same files each time)

gildas-lormeau commented 5 months ago

Can you reproduce the issue if you replace new WritableStream({ ... }) with a writableproperty from a TransformStream object? In that case, is the readable property of the TransformStream returning a corrupted stream?

DebianArch64 commented 5 months ago

Same thing happens

const {readable, writable} = new TransformStream({flush: () => { writer.close() }})
  const reader = readable.getReader()
  const yes = async () => {
  while (true) {
    const { done, value } = await reader.read();
    if (done) {
      break;
    }

    // send chunk
  }
  yes()
  writer.readable.pipeTo(writable, { preventClose: true })

Output without delay:

  Archive:  /Users/debianarch/Downloads/install (18).ipa
  inflating: Payload/Taurine.app/essential_0-4_iphoneos-arm.deb  
  inflating: Payload/Taurine.app/bootstrap.tar.gz  
  error:  invalid compressed data to inflate
file #3:  bad zipfile offset (local header sig):  12594496
file #4:  bad zipfile offset (local header sig):  12603319
file #5:  bad zipfile offset (local header sig):  12617048
file #6:  bad zipfile offset (local header sig):  15534336
file #7:  bad zipfile offset (local header sig):  15535204
file #8:  bad zipfile offset (local header sig):  15535920
file #9:  bad zipfile offset (local header sig):  15536177
file #10:  bad zipfile offset (local header sig):  15776964
file #11:  bad zipfile offset (local header sig):  15777490
file #12:  bad zipfile offset (local header sig):  15779564
file #13:  bad zipfile offset (local header sig):  15780306
file #14:  bad zipfile offset (local header sig):  15797892
file #15:  bad zipfile offset (local header sig):  15799206
file #16:  bad zipfile offset (local header sig):  15799489
file #17:  bad zipfile offset (local header sig):  28942874
file #18:  bad zipfile offset (local header sig):  28961940
file #19:  bad zipfile offset (local header sig):  28968084
  inflating: Payload/Taurine.app/signcert.p12  
  inflating: Payload/Taurine.app/Info.plist  
  inflating: Payload/Taurine.app/PkgInfo  
  inflating: Payload/Taurine.app/_CodeSignature/CodeResources  
  inflating: Payload/Taurine.app/Taurine
DebianArch64 commented 5 months ago

My bad found the issue, i'm embarrassed sorry for wasting your time

gildas-lormeau commented 5 months ago

No problem, I'm glad to hear that the issue is fixed :)