Support java.nio.file Paths and FileSystems with nested jars

[dsyer]

In Spring Boot 1.3.x this code works if "/mydir" is in the parent archive (i.e. src/main/resources in the project that creates the jar):

Resource resource = new ClassPathResource("/mydir");
Paths.get(resource.getURI());

In 1.4.x it throws FileSystemNotFoundException which isn't even an IOException, so it breaks existing apps.

[dsyer]

Some code: https://github.com/spring-projects/spring-boot-issues/tree/master/gh-7161

[wilkinsona]

I get a FileSystemNotFoundException with 1.3.8 as well:

jar:file:/Users/awilkinson/dev/spring/spring-boot-issues/gh-7161/target/demo-0.0.1-SNAPSHOT.jar!/mydir
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:54)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:104)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:61)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)
Caused by: java.nio.file.FileSystemNotFoundException
    at com.sun.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:171)
    at com.sun.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:157)
    at java.nio.file.Paths.get(Paths.java:143)
    at com.example.SimpleApplication.main(SimpleApplication.java:17)
    ... 8 more
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:62)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:104)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:61)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:54)
    ... 3 more
Caused by: java.nio.file.FileSystemNotFoundException
    at com.sun.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:171)
    at com.sun.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:157)
    at java.nio.file.Paths.get(Paths.java:143)
    at com.example.SimpleApplication.main(SimpleApplication.java:17)
    ... 8 more

[dsyer]

Hmm. Me too, but I thought I tried a "real" use case and 1.3.8 fixed it. Odd. Does that mean we can't fix it in Boot? It's to do with the URI that comes back from Resource.getURI().

[dsyer]

The stack trace is slightly different in 1.4. Maybe that was enough to fix my real use case:

Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:58)
Caused by: java.nio.file.FileSystemNotFoundException
    at com.sun.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:171)
    at com.sun.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:157)
    at java.nio.file.Paths.get(Paths.java:143)
    at com.example.SimpleApplication.main(SimpleApplication.java:18)
    ... 8 more

[wilkinsona]

ZipFileSystemProvider is used because it handles URLs with a jar scheme. It turns the URI (jar:file:/Users/awilkinson/dev/spring/spring-boot-issues/gh-7161/target/demo-0.0.1-SNAPSHOT.jar!/mydir) into a Path (/Users/awilkinson/dev/spring/spring-boot-issues/gh-7161/target/demo-0.0.1-SNAPSHOT.jar) which is used to look up a FileSystem in a Map. There's no FileSystem for the Path so the FileSystemNotFoundException is thrown.

If you create a new FileSystem for the URI before trying to get a path, it works with both 1.3.8 and 1.4:

package com.example;

import java.io.IOException;
import java.net.URI;
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Collections;

import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.io.Resource;

@SpringBootApplication
public class SimpleApplication {

    public static void main(String[] args) throws IOException {
        Resource resource = new ClassPathResource("/mydir");
        URI uri = resource.getURI();
        System.out.println(uri);
        FileSystems.newFileSystem(uri, Collections.emptyMap());
        Path path = Paths.get(uri);
        System.out.println(path);
    }

}

[dsyer]

Interesting. It doesn't fix my legacy app yet though, because it still expects either an IOException, or a valid file system.

[wilkinsona]

@dsyer Can you share your legacy app, or something that reproduces its behaviour?

[dsyer]

I updated the sample adding some features. It now fails on startup because a ZipFileSystem doesn't support watches. I'm quite happy to fix the "legacy" code and make it less sensitive to exceptions. But it feels to me like using Paths with ueberjars is not a daft thing to do, and it's hard, so maybe we can make it easier if we are creative.

[166MMX]

I'd rather consider this issue a bug then an enhancement. Especially since at least 2 further issues are related to this one.

The workaround I use is quite ugly by converting the URI to a string, replace a character sequence and creating a new URI with the patched string.

[bdnett]

Here's an example program that gets an exception with the Files.walk method:

package com.example;

import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.io.Resource;

import java.io.IOException;
import java.net.URI;
import java.nio.file.*;
import java.util.Collections;

@SpringBootApplication
public class SimpleApplication {

    public static void main(String[] args) throws IOException {
        Resource resource = new ClassPathResource("/mydir");
        URI uri = resource.getURI();
        System.out.println(uri);
        FileSystem fs = FileSystems.newFileSystem(uri, Collections.emptyMap());
        Path path = Paths.get(uri);
        System.out.println(path);
        if (Files.isDirectory(path)) {
            System.out.println("file type is directory");
        } else if (Files.isRegularFile(path)) {
            System.out.println("file type is regular file");
        } else {
            System.out.println("file is neither directory nor regular");
        }
        Files.walk(path).forEach((Path child) -> {
            if (Files.isRegularFile(child)) {
                System.out.print("Regular file: " + child);
            } else {
                System.out.print("Directory: " + child);
            }
        });

    }

}

And I added some directories and files to the jar file (contents not important for example):

        0  2017-04-24 16:28   BOOT-INF/classes/mydir/
        0  2017-04-24 16:28   BOOT-INF/classes/mydir/mySubDir/
       13  2017-04-24 16:28   BOOT-INF/classes/mydir/mySubDir/Hello.txt

[Caratacus]

Also encountered this problem, continued attention.

[edwardsre]

Encountered this problem also.

[KingstonDuo]

I too am encountering this issue. I am trying to get the file path to build up a command line command to execute and when run as a fat jar I am getting the FileSystemNotFoundException.

[neterium]

Same issue here :(

[sgrimm-sg]

I also ran into this. My workaround is to explicitly prefix the path with BOOT-INF/classes like so:

  String resourceDirectory = "/my/resource/dir";
  URI uri = getClass().getResource(resourceDirectory).toURI();
  Path path;

  if (uri.getScheme().equals("jar")) {
    FileSystem fileSystem = FileSystems.newFileSystem(uri, Collections.emptyMap());
    path = fileSystem.getPath("/BOOT-INF/classes" + resourceDirectory);
  } else {
    // Not running in a jar, so just use a regular filesystem path
    path = Paths.get(uri);
  }

  Files.walk(path).forEach(...);

That works but I really don't like having an implementation detail of Spring Boot's executable jar layout hardwired into my code.

[unmarshall]

I use springboot 1.5.9.RELEASE and i still get this exception. I am trying to read a resource which is packaged in BOOT-INF/classes/someDir/file. I have the following code

final URI resourceURI = new ClassPathResource("/my/resource");
FileSystems.newFileSystem(resourceURI, Collections.emptyMap());
final byte[] bytes = Files.readAllBytes(resourceURI);

It resolves the URI to BOOT-INF/classes!/my/resource but it throws a java.nio.file.NoSuchFileException. I checked and the file is very much there.

I am a bit surprised that this issue has been open since 2016. Any resolutions?

Best Regards, Madhav

[dsyer]

The example above doesn't compile (ClassPathResource is not a URI), so it's probably not a real use case?

There is no need to use Files here. You could, for instance, just do this:

final byte[] bytes = StreamUtils.copyToByteArray(new ClassPathResource("/my/resource").getInputStream());

[osamaabdulsattar]

@dsyer This is the only way that worked for me

[166MMX]

@dsyer Over a year ago I was repacking jRuby and needed to test whether a given resource is a regular file or a directory or exists at all. I documented everything over #8822 and created a pull request https://github.com/spring-projects/spring-boot-issues/pull/66 . Hopefully that use case is real enough to work for you.

[arpan2501]

Hey folks any guidance on this question

:: Spring Boot :: (v2.0.3.RELEASE)

application.properties:

spring.cloud.gcp.credentials.location=classpath:ArpanShoppingApp-863d536d1f93.json

and running jar file gives exception

java -jar CloudSQLConnect-1.0.jar

2018-06-22 10:46:38.393  INFO 1172 --- [           main] o.s.c.g.s.a.GcpCloudSqlAutoConfiguration : Default MYSQL JdbcUrl provider. Connecting to jdbc:mysql://google/google_sql?cloudSqlInstance=mindful-highway-207309:asia-south1:shopping-db&socketFactory=com.google.cloud.sql.mysql.SocketFactory&useSSL=false with driver com.mysql.jdbc.Driver
2018-06-22 10:46:38.401  INFO 1172 --- [           main] o.s.c.g.s.a.GcpCloudSqlAutoConfiguration : Error reading Cloud SQL credentials file.

java.io.FileNotFoundException: class path resource [ArpanShoppingApp-863d536d1f93.json] cannot be resolved to absolute file path because it does not reside in the file system: jar:file:/Users/arpan/Documents/workspace-sts-3.8.4.RELEASE/CloudSQLConnect/target/CloudSQLConnect-1.0.jar!/BOOT-INF/classes!/ArpanShoppingApp-863d536d1f93.json
at org.springframework.util.ResourceUtils.getFile(ResourceUtils.java:217) ~[spring-core-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.core.io.AbstractFileResolvingResource.getFile(AbstractFileResolvingResource.java:133) ~[spring-core-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.cloud.gcp.sql.autoconfig.GcpCloudSqlAutoConfiguration.setCredentialsProperty(GcpCloudSqlAutoConfiguration.java:167) [spring-cloud-gcp-starter-sql-1.0.0.M1.jar!/:1.0.0.M1]
at org.springframework.cloud.gcp.sql.autoconfig.GcpCloudSqlAutoConfiguration.defaultJdbcInfoProvider(GcpCloudSqlAutoConfiguration.java:107) [spring-cloud-gcp-starter-sql-1.0.0.M1.jar!/:1.0.0.M1]
at org.springframework.cloud.gcp.sql.autoconfig.GcpCloudSqlAutoConfiguration$$EnhancerBySpringCGLIB$$edf77794.CGLIB$defaultJdbcInfoProvider$1(<generated>) [spring-cloud-gcp-starter-sql-1.0.0.M1.jar!/:1.0.0.M1]

[wilkinsona]

@arpan2501 That doesn't appear to be related to this issue as the code in the stack isn't using Paths or FileSystem. If you're looking for some help about Spring Cloud GCP (which is where the problem appears to be), please ask a question on Stack Overflow.

[arpan2501]

@wilkinsona I thought it might be related to Paths because I am able to connect to Cloud SQL perfectly when running the Spring Boot App. The issue comes only when I build and try to run the Jar and it complains FileNotFoundException with this ! in path target/CloudSQLConnect-1.0.jar!/BOOT-INF/classes!/ArpanShoppingApp-863d536d1f93.json

Sorry to bother you here.. Raised the same in stackoverflow!!!

[magneticflux-]

I did some work on a similar issue to this here: https://github.com/magneticflux-/classpath-resource-extractor/issues/2 and https://github.com/magneticflux-/classpath-resource-extractor/pull/3

It has a method that visits a URI (that may be nested (including Spring's broken URIs)) as a Path.

[ermicioi]

Related issue appear when @ConfigurationProperties class with java.nio.file.Path property is used.

When such application run from Intellj all works OK, because PathEditor uses ClassLoaders$AppClassLoader however when run from fat-jar which uses LaunchedURLClassLoader it fails.

Some more information at https://stackoverflow.com/questions/64768787/spring-boot-property-of-type-path-read-from-application-yaml

Spring Boot version: 2.3.5.RELEASE

[wilkinsona]

Prompted by trying to upgrade to Jetty 12 and its increased reliance upon Path, I've been looking at this again. Unfortunately, I've not found much to give me hope that it can be fixed in a transparent manner. Unlike with jar: URLs and our custom URLStreamHandler, we can't plug in a custom file system for jar: URLs as the JDK's ZipFileSystemProvider always takes precedence for URIs with a jar scheme.

I also can't see a way to get things working with the built-in provider. Using jar:jar:file URIs as @magneticflux- has done fails when calling FileSystems.newFileSystem(uri, Collections.emptyMap()), possibly due to a JDK bug in how the spec is stripped off. You can work around that, but it requires calling some custom code that prevents any support from being transparent. Using jar:file URIs as we currently do works up until someone tries to read the Path at which points it fails with a java.nio.file.NoSuchFileException.

The only path forward that I can see for FileSystem support that is fully in our control would be to use a custom scheme such as bootjar: for the URI which would then allow us to plug in our own FileSystemProvider that wouldn't be clobbered by the JDK's defaults. Unfortunately, I think this would break a prohibitively large amount of existing code that works just fine with our jar: URLs as they wouldn't know how to deal with the custom bootjar: scheme.

[steve-t7]

Hello so I run through the same problem at work, it took severals steps to find the solution as I cannot reproduce it in a local environment. This problem only occured in a cloud environment, I reproduce it by building a little app in a docker with the same caracteristics as my feature.

The problem was that I was accessing resources from another jar which was a dependecy of my deployed jar, so when I tried accessing the resource, the uri scheme was jar: and as it is stated ZipFileSystemProvider takes precedence.

My solution was to not work with files but with inputStream, they are many ways to do it, I've used Resources.getResource(...).openStream() from guava it works well to get InputStream, and in my case it was enough