search

hashicorp / packer

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.
http://www.packer.io
Other
15.05k stars 3.32k forks source link

Zipping files 5.5GB or larger can cause incorrect headers to be put into the zip archive #7889

Open darwin-userful opened 5 years ago

darwin-userful commented 5 years ago

Packer Version: Confirmed in 1.3.5, 1.4.0, 1.4.2 Host Platform: Windows Server 2012

Debug log and the template used.: https://gist.github.com/darwin-userful/11592faa73653612c32d1b79749f06bc

STEPS TO REPRODUCE:

  1. Find a large file to compress. The example template uses the Windows Server Data Center Evaluation ISO, but I suspect any file larger then 5.5Gb will suffice; try 6Gb to be safe. Alter the template.json to match if using a different file.

2) Run packer build template.json

WHAT SHOULD HAPPEN: The outputted .zip file should be unzippable without complaint with either 7zip or the Powershell unzipping cmdlets.

WHAT ACTUALLY HAPPENS: If you use 7zip to test the zip file, it will warn of a headers error. ~If attempting to extract using Windows powershell, it will throw an exception with a similar complaint about the headers.~ However you can still extract the archive AFAIK.

BACKGROUND: We're using Packer to build vhdx files. In the past there were no problems as the produced images were around 5GB prior to zipping, however a recent change made the vhdx slightly larger which appears to trip this problem (the image files we use are around 5.4GB but I'm quoting off the top of my head). The main issue is that another part of the process uses powershell to extract the created archive, and powershell will throw an exception if said headers are not correct.

We've worked around the issue for the time being by calling 7zip to create the archive, so there isn't a hurry to get this fixed. But I didn't see any open issues for this and thought that you might like to be aware of it.

SwampDragons commented 5 years ago

I'm having trouble reproducing this. what is your command for extracting the archives?

darwin-userful commented 5 years ago

Odd, I just reproduced it no problem using a Windows 10 as the packer platform and Packer 1.4.2; 7zip test complains that the header is wrong.

I'll get the the Server 2012 expand command in a second, it got buried somewhere and 2012 doesn't have that nice expansion command. Windows 10 and Server 2016 however have a shortcut cmdlet which is also exhibiting a strange issue:

Expand-Archive .\Windows_Server_2016_Datacenter_EVAL_en-us_14393_refresh.ISO.zip -D estinationPath .\

When this command is run, it executes without any errors. However the extracted ISO file is only 4GB, whereas the original ISO is 6.5GB

Winrar expands it without complaint FWIW.

darwin-userful commented 5 years ago

Scratch what I said about the throwing an exception part, I just found out they're not using that method anyways (edited original report to reflect this). The Extract-Archive cmdlet is what they're now using, although the file being wrong size is... almost as bad a problem if not worse; thrown exceptions tell you right off the bat something's wrong, a mangled filed doesn't get detected until further down the pipeline.

Are you using the ISO I tested with or a different file? If a different file, would it help if I uploaded a file that I can reproduce the problem with to somewhere?

SwampDragons commented 5 years ago

I'd generated a test file rather than searching out and downloading the windows one; all the windows ISOs I have stored locally are < 5 GB. But knowing you weren't getting an exception helps. I was using Expand-Archive, too. I'll find an iso that I can use to test this more realistically, no need to send me one. :)