Support zip64 format executable archives

[pmdusso]

I first wrote this in the Gradle forum, and they recommended me transfer it to here.

I wasn't able to execute the war generated from bootRepackage in a project derived from jHipster framework. The topic still opened in Stackoverflow.

I created a small application and put it on Github to help you help me with this. Following these steps I believe you can reproduce the problem also. I'm first packing a simple Spring application and then running the generated war. This will create 66000 files in the resource folder. Packing the same app again will fail because in the second time there is more than 65535 files. We confirmed this by cleaning the resource folder and bootRepackaging it again successfully.

Then I set the zip64 property to true (I had to do it in the jar { } task also. In my original application however I do not have a jar task and set only in the war task). The bootRepackage task completes with success, but then when I run the war I get the no Start-Class error.

Running this for the first time will work, because total files < 65535 and zip64 = false

./gradlew clean bootRepackage; java -jar build/libs/app-0.1-SNAPSHOT.war

Running this for the second time will not work, because total files > 65535 and zip64 = false

./gradlew clean bootRepackage; java -jar build/libs/app-0.1-SNAPSHOT.war

Running this for the third time (equal to alternative one) will work, because total files < 65535 and zip64 = false

rm src/main/resources/*; ./gradlew clean bootRepackage; java -jar build/libs/app-0.1-SNAPSHOT.war

At this point we have total files > 65535 and zip64 = false. We change zip64 property to true and try to bootPackage it again

Set zip64 = true in build.gradle in jar and war tasks. ./gradlew clean bootRepackage; java -jar build/libs/app-0.1-SNAPSHOT.war

java.lang.IllegalStateException: No 'Start-Class' manifest entry specified in jar:file:../boot-repackage-test/build/libs/app-0.1-SNAPSHOT.war!/
        at org.springframework.boot.loader.archive.Archive.getMainClass(Archive.java:57)
        at org.springframework.boot.loader.ExecutableArchiveLauncher.getMainClass(ExecutableArchiveLauncher.java:69)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:61)
        at org.springframework.boot.loader.WarLauncher.main(WarLauncher.java:61)

To conclude, unzip app-0.1-SNAPSHOT.war && cat META-INF/MANIFEST.MF yields:

Manifest-Version: 1.0
Start-Class: org.Application
Spring-Boot-Version: 1.2.3.RELEASE
Main-Class: org.springframework.boot.loader.WarLauncher

In gradle forum they pointed the following class: spring-projects/spring-boot/blob/master/spring-boot-tools/spring-boot-loader/src/main/java/org/springframework/boot/loader/jar/CentralDirectoryEndRecord.java

Best regards, Pedro Dusso

[wilkinsona]

This is a known limitation; Spring Boot's Loader doesn't support the zip64 format. Given the support for nested jars, why do you need more than 65535 files in your executable archive?

[pmdusso]

I don't know about the nested jars. Can you give point any documentation? I would like to try that. May scenario is a Spring web application with java in back and angular in the front. I guess the excess of files comes from the libraries installed to the frontend.

[wilkinsona]

Nested jars are documented here.

Even with a web application, I'm surprised at the need for more than 65535 files in your archive. Could you use something like minify to reduce the number of files (and speed things up a bit for your users)?

[tenstriker]

Hi @wilkinsona since it's a limitation of spring-boot classloader It wouldn't be fixed if I use latest jre 7 or jre 8. Correct me if I'm wrong there. Interestingly it works when I run it from eclipse using same JRE! My eclipse project has maven dependencies on classpath. I assume eclipse internally uses java -cp option instead of java -jar option. SO is there a way from command-line to run spring-boot jar using java -cp option. Note: as I mentioned in case #5086 I use PropertyLauncher. May be I can try Jar launcher?

[wilkinsona]

since it's a limitation of spring-boot classloader

Just to avoid any confusion, tt's not a limitation of Spring Boot's ClassLoader. It's a limitation of Spring Boot's Loader which is used by the various launchers to read zip archives.

It wouldn't be fixed if I use latest jre 7 or jre 8. Correct me if I'm wrong there.

Correct. Changing the version of the JRE will make no difference.

Interestingly it works when I run it from eclipse using same JRE!

When you run your application's main method in Eclipse, none of Spring Boot's launchers is used so you avoid the problem.

is there a way from command-line to run spring-boot jar using java -cp option. Note: as I mentioned in case #5086 I use PropertyLauncher. May be I can try Jar launcher?

Using JarLauncher won't make any difference as all of the launchers use the same code for reading zip archives. You could work around the problem by not repackaging your application into an executable archive, building the classpath yourself, and calling your application's main method directly.

[tenstriker]

To circumvent this issue, I manually extracted all the zip64 jars and pointed that directory to spring-boot loader. However, this is a dirty solution. I think it's really worthwhile for spring-boot loader to support zip64 files. Java 1.7 and above by default uses zip64 based on file size. There is no way to convert third-party zip64 to regular zip file and it's also seems counter intuitive.

[philwebb]

@tenstriker Unfortunately supporting zip64 is quite problematic for us since it makes it much harder to detect the embedded script that we put at the front of the file. What build tool are you using? zip64 should really only be needed if there are a large number of files in the archive (the size of the files should not matter).

[tenstriker]

we use maven scala and java compiler and spring-boot-maven-plugin plugins. As I mentioned in earlier comment zip64 jar is provided by third-party and we don't have control over it. We just have to load it in our spring boot app during runtime. Probably that zip file does has large number of files.

[tan9]

I have a project using org.webjars.npm:material-design-icons:jar:3.0.1 which containing 89824 files has the same problem.

[abhijeettannu]

So what is the alternative to this issue. I am working on an application and it dependent on a third party jar which is needed by the application. This third party jar has more than 65535 files and so Spring Boot Loader is not accepting it and failing to start the application. What other options we have? Can we break a fat jar into multiple jars?

[abhijeettannu]

@tan9 - so what did you do to fix it?

[abhijeettannu]

@philwebb - please suggest

[tan9]

@abhijeettannu, I removed the jar, and found an alternative that won't exceed the limit. :pensive:

[philwebb]

@abhijeettannu I'm afraid I don't have an answer for you. Splitting up your jar, or trying the shade plugin are the only real options at the moment. Out of interest, what is the third-party jar that you're using?

[abhijeettannu]

@philwebb - but won't the shading cause more files being added to the jar and making it more fat?

[philwebb]

@abhijeettannu It will, but it means you can use a standard jar (everything is unpacked, no nested jars) so you don't need to use our JarLoader mechanism at all.

[abhijeettannu]

Hi @philwebb - i did the unpacking of all jars into single spring boot jar. This single jar didn't had any nested jars. But still while running the app using "java -jar abc.jar" i still get same exception. I think the spring loader still checks number of files in this single jar, which in my case are more than 65535 as the fat jar was unpacked in this single jar.

Caused by: java.lang.IllegalStateException: Zip64 archives are not supported
    at org.springframework.boot.loader.jar.CentralDirectoryEndRecord.getNumberOfRecords(CentralDirectoryEndRecord.java:124)

Code from spring boot loader:

public int getNumberOfRecords() {
        long numberOfRecords = Bytes.littleEndianValue(this.block, this.offset + 10, 2);
        if (numberOfRecords == 65535L) {
            throw new IllegalStateException("Zip64 archives are not supported");
        } else {
            return (int)numberOfRecords;
        }
    }

[abhijeettannu]

@philwebb - did you meant to create a normal jar (non spring boot) and have unpacked jars inside it and inside manifest mention the starter class?

[philwebb]

@abhijeettannu Disable the boot repackaging or remove the Spring Boot Maven plugin.

[lygoing]

@philwebb Hi. I also encountered the same problem. Is there a solution to this problem now?

[lygoing]

org.webjars.npm:material-design-icons:jar:3.0.1

[philwebb]

@lygoing The issue is still open I'm afraid.

[emulvihill]

This is now affecting me as well. Hopefully there is a solution on the horizon.

[nicmaster]

Hi All, I also encountered the same problem. Is there a solution to this problem now?

[wilkinsona]

@nicmaster No, sorry. This issue remains open.

[cvienot]

I submitted a pull request to support zip64 jars. I added a test that successfully scan the 65537 files. However it might not be enough for such a change. Any feedback is welcome

[johnhunt]

So I spent a lot of last week pulling my hair out over this issue. We need to include the groupdocs conversion package in our jar file, which appears to be impossible because of the file limit. We ended up being unable to do this, can anyone explain what workarounds there are? Maybe they're aren't any, perhaps we hit a brick wall?

[zapphyre]

@cvienot, how can I test your change? Is it available in some spring boot release? Had no idea of this limitation..

[gmakin]

As a workaround, I switched to use maven shade plugin. It worked!

[charany1]

@gmakin using shade-plugin resolves this zip64 issue but gives : no main manifest attribute,

whereas on exploding the jar and looking upon contents of META-INF/MANIFEST.mf , I can that there is a Main-Class specified , any idea ?

[snicoll]

@charany1 please use StackOverflow to ask questions.

[wilkinsona]

Closing in favour of #16091.

[yasin0824github]

finally i resolve this problem,,,i'm use idea 2020 version.....