Support prefix bytes with fat jars

[aantono]

When trying to create a self-starting bash script with embedded jar (https://coderwall.com/p/ssuaxa) the startup throws an exception.

Exception in thread "main" java.lang.IllegalStateException: java.io.IOException: Unable to read bytes
    at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLauncher.java:52)
    at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLauncher.java:44)
    at org.springframework.boot.loader.JarLauncher.<init>(JarLauncher.java:30)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:45)
Caused by: java.io.IOException: Unable to read bytes
    at org.springframework.boot.loader.jar.Bytes.get(Bytes.java:50)
    at org.springframework.boot.loader.jar.JarEntryData.<init>(JarEntryData.java:66)
    at org.springframework.boot.loader.jar.JarEntryData.fromInputStream(JarEntryData.java:170)
    at org.springframework.boot.loader.jar.JarFile.loadJarEntries(JarFile.java:136)
    at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:126)
    at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:107)
    at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:97)
    at org.springframework.boot.loader.archive.JarFileArchive.<init>(JarFileArchive.java:47)
    at org.springframework.boot.loader.Launcher.createArchive(Launcher.java:154)
    at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLauncher.java:49)
    ... 3 more

When testing the file with jar tvf run.sh the content of the file gets printed just fine.

After reading http://mesosphere.io/2013/12/07/executable-jars/ it appears that the Zip/Jar format supports prepended arbitrary, non-Zip data and yet remain interpretable to the Zip decompressor, which is what is being used to create self-executing applications.

For some reason the Bytes parsing errors when trying to read that kind of jar. :(

[philwebb]

Interesting technique, I've not seen that before. Spring Boot has it's own Zip Processing logic which it uses to work out how to seek inside nested jars. My guess is that our logic incorrectly assumes that zip data actually starts at position 0, where as these files have additional bytes at the front.

If your interested in the gory details you can read this http://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#executable-jar and looks at the spring-boot-loader project.

I'll try to have a look and see if we can make this work. It would actually be really nice feature to add to the Maven/Gradle plugins.

Thanks for the link!

[philwebb]

Try the latest 1.1.2.BUILD-SNAPSHOT, I think this should work now.

[aantono]

Phil,

I just tried the snapshot build, but still get an error. :(

Exception in thread "main" java.lang.IllegalStateException: java.io.IOException: Unable to read bytes
    at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLauncher.java:52)
    at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLauncher.java:44)
    at org.springframework.boot.loader.JarLauncher.<init>(JarLauncher.java:30)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:45)
Caused by: java.io.IOException: Unable to read bytes
    at org.springframework.boot.loader.jar.Bytes.get(Bytes.java:50)
    at org.springframework.boot.loader.jar.JarEntryData.<init>(JarEntryData.java:66)
    at org.springframework.boot.loader.jar.JarEntryData.fromInputStream(JarEntryData.java:170)
    at org.springframework.boot.loader.jar.JarFile.loadJarEntries(JarFile.java:136)
    at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:126)
    at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:107)
    at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:97)
    at org.springframework.boot.loader.archive.JarFileArchive.<init>(JarFileArchive.java:47)
    at org.springframework.boot.loader.Launcher.createArchive(Launcher.java:154)
    at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLauncher.java:49)
    ... 3 more

Here is the little groovy snippet I did to scan the InputStream which forwards the offset to the beginning of the archive data:

byte[] HA = [0x50, 0x4b, 0x03, 0x04] as byte[] //80, 75, 3, 4 -> "PK\003\004"

def findFirst(PushbackInputStream is) {
    int b = is.read()
    println pbs.available()
    println "Just read " + b
    if (b != -1) {
        if(HA[0] == b) {
            println("Found first - " + b)
            println pbs.available()
            return readNext(is, 1)
        } else {
            println("Searching next first - " + b)
            println pbs.available()
            return findFirst(is)
        }
    } 
    return false
}
def readNext(PushbackInputStream is, idx) {
    int b = is.read()
    if (b != -1) {
        if(idx < HA.length) {
            if(HA[idx] == b) {
                println("Found next " + idx + " - " + b)
                println pbs.available()
                return readNext(is, idx + 1)
            } else {
                println("Unread next " + idx + " - " + b)
                is.unread(b)
                println pbs.available()
                return findFirst(is)
            }
        } else {
            println("Found Header at " + idx + " - " + b)
            println pbs.available()
            is.unread(b)
            is.unread(HA)
            println pbs.available()
            return true
        }
    } else {
        println("EOF")
    }
    return false
}

fis = new FileInputStream("test.run")
pbs = new PushbackInputStream(fis, 5)
found = findFirst(pbs)
println((found ? "Found" : "Not Found") + " Zip Header")
zis = new java.util.zip.ZipInputStream(pbs)
println zis.getNextEntry()

The approach I used to generate the test.run is the following:

gradlew build
cat stub.sh build/libs/spring-boot-sandbox.jar > test.run
chmod +x test.run
./test.run

The content of the stub.sh is:

#!/bin/sh
MYSELF=`which "$0" 2>/dev/null`
[ $? -gt 0 -a -f "$0" ] && MYSELF="./$0"
java=java
if test -n "$JAVA_HOME"; then
    java="$JAVA_HOME/bin/java"
fi
exec "$java" $java_args -jar $MYSELF "$@"
exit 1

[aantono]

So I looked into the Zip file format and based on http://en.wikipedia.org/wiki/Zip_(file_format)#File_headers and http://www.pkware.com/documents/casestudies/APPNOTE.TXT (Section 4.3.7) The starting of the zip content is marked by the PK/0x04034b50 (read as a little-endian number). So maybe start reading the byte stream until encountering the first marker and start parsing from there?

The main reason I started looking into this solution is to be able to create shell scripts that can have some logic to setup the java command line, mostly to handle the jvm -D system properties and memory/gc/etc settings, as it is, unfortunately, not possible to declare those from within the jar. :(

[philwebb]

We have some problems with our infrastructure at the moment. The CI server is down due to a data center outage. It's possible that the latest SNAPSHOT doesn't include my fix.

I just tried a local build and your stub.sh script prefixed to the front worked for me.

In the end I didn't need to search for the header bytes because the start of the actual ZIP data can be calculated (see https://github.com/spring-projects/spring-boot/blob/54dc46f7773aac5fab0d48b1aa861b4a0ad5b7c3/spring-boot-tools/spring-boot-loader/src/main/java/org/springframework/boot/loader/jar/CentralDirectoryEndRecord.java#L97)

[aantono]

Oh, great find, didn't think of that... I just pulled master branch and did a local publish, so verified that it all works and starts. Would be interested to discuss/see where you guys are thinking of taking this possible route to make self-starting Linux/Mac executables (I'm trying to figure out if there might be an option to somehow extract the Manifest/prop file from the main jar and use it to set the JVM startup command line arguments, things like XML parser providers, security providers, memory settings, etc, things that can't be set from within via System.setProperty() way.

[philwebb]

So far, my initial thoughts were that we could potentially have the Maven or Gradle plugins include the script for you (or let you pick one). Initially I was thinking about using the script to support #519, perhaps providing a script that has start, stop, status etc. so that you could install a boot app as a service just by using a symlink to etc/rc3.d.

I hadn't actually considered some of the other use cases that you bring up, they would be an interesting addition. My bash skills aren't exactly great so anything you can share would be much appreciated.

[philwebb]

@aantono Feel free to open another issue if there are specific enhancements that we should be thinking about.

[Robin479]

Hi all. I haven't looked at the details of the init.d features, yet, but I wanted to mention here that there is more to a daemon than a simple embedded init.d interface. Specifically, a daemon should (be able to) drop superuser privileges once it has consumed boot configuration files and acquired protected system resources (e.g. binding to port 80). The "user code" that makes up the actual spring application should then be initialized and run without such privileges, e.g. file system IO should usually be restricted to (service) user file permissions (e.g. during usual operation, a newly created file belongs to "myappsysuser" rather than root).

This would certainly require a two-stage ApplicationContext: initializing Spring Boot, acquiring privileged resource, dropping the JVM process to user privileges (a native system call), and then initializing the user coded/provided/configured subsystems. Have a look at commons-daemon's Daemon interface: initialize()/shutdown() is where you should read system configuration files and acquire privileged resources as root, and start()/stop() is where you actually startup the user's application including thread pools and database access.

[Robin479]

...by "user" I mean "developer building a Spring Boot integrated application", of course.