mobie / mobie-io

BSD 2-Clause "Simplified" License
4 stars 8 forks source link

cannot read ome-zarr from s3 #138

Open d-v-b opened 1 year ago

d-v-b commented 1 year ago

Attempting to load the following dataset from s3 (s3://janelia-cosem-datasets/jrc_hela-2/jrc_hela-2.zarr/em/fibsem-uint16/) causes the ome-zarr bdv plugin to fail. Unfortunately the plugin doesn't record the full exception traceback, but here's what appeared:

traceback ``` (Fiji Is Just) ImageJ 2.9.0/1.53t; Java 1.8.0_322 [64-bit]; Mac OS X 10.16; 100MB of 2868MB (3%) java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:308) at net.imagej.legacy.DefaultLegacyHooks.interceptRunPlugIn(DefaultLegacyHooks.java:166) at ij.IJ.runPlugIn(IJ.java) at ij.Executer.runCommand(Executer.java:152) at ij.Executer.run(Executer.java:70) at java.lang.Thread.run(Thread.java:750) Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:304) ... 5 more Caused by: java.lang.RuntimeException: Module threw exception at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127) at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63) at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: null at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008) at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1416) at org.embl.mobie.io.util.S3Utils.getS3Client(S3Utils.java:78) at org.embl.mobie.io.ome.zarr.loaders.N5S3OMEZarrImageLoader$N5S3ZarrReaderCreator.create(N5S3OMEZarrImageLoader.java:93) at org.embl.mobie.io.ome.zarr.loaders.N5S3OMEZarrImageLoader.(N5S3OMEZarrImageLoader.java:60) at org.embl.mobie.io.ome.zarr.openers.OMEZarrS3Opener.readKey(OMEZarrS3Opener.java:70) at org.embl.mobie.io.ome.zarr.openers.OMEZarrS3Opener.readURL(OMEZarrS3Opener.java:53) at org.embl.mobie.command.OpenOMEZARRFromS3Command.openAndShow(OpenOMEZARRFromS3Command.java:53) at org.embl.mobie.command.OpenOMEZARRFromS3Command.run(OpenOMEZARRFromS3Command.java:62) at org.scijava.command.CommandModule.run(CommandModule.java:196) at org.scijava.module.ModuleRunner.run(ModuleRunner.java:163) at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124) ... 6 more Caused by: org.apache.http.client.ClientProtocolException at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1330) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ... 26 more Caused by: org.apache.http.HttpException: s3 protocol is not supported at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:89) at org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ... 31 more ```

The multiscale metadata for that dataset is here:

.zattrs ```json { "multiscales": [ { "axes": [ { "name": "z", "type": "space", "unit": "nanometer" }, { "name": "y", "type": "space", "unit": "nanometer" }, { "name": "x", "type": "space", "unit": "nanometer" } ], "coordinateTransformations": [ { "scale": [ 1.0, 1.0, 1.0 ], "type": "scale" } ], "datasets": [ { "coordinateTransformations": [ { "scale": [ 5.24, 4.0, 4.0 ], "type": "scale" }, { "translation": [ 0.0, 0.0, 0.0 ], "type": "translation" } ], "path": "s0" }, { "coordinateTransformations": [ { "scale": [ 10.48, 8.0, 8.0 ], "type": "scale" }, { "translation": [ 2.62, 2.0, 2.0 ], "type": "translation" } ], "path": "s1" }, { "coordinateTransformations": [ { "scale": [ 20.960000000000004, 16.0, 16.0 ], "type": "scale" }, { "translation": [ 7.86, 6.0, 6.0 ], "type": "translation" } ], "path": "s2" }, { "coordinateTransformations": [ { "scale": [ 41.92, 32.0, 32.0 ], "type": "scale" }, { "translation": [ 18.340000000000003, 14.0, 14.0 ], "type": "translation" } ], "path": "s3" }, { "coordinateTransformations": [ { "scale": [ 83.84, 64.0, 64.0 ], "type": "scale" }, { "translation": [ 39.300000000000004, 30.0, 30.0 ], "type": "translation" } ], "path": "s4" } ], "metadata": null, "name": "", "type": null, "version": "0.4" } ] } ```
tischi commented 1 year ago

Dear @d-v-b ,

Thank you for reporting this! This seems like some S3 error: Caused by: org.apache.http.HttpException: s3 protocol is not supported Not sure what is going on...

@constantinpape can you maybe try opening the above data set using some python API to see if that works?

d-v-b commented 1 year ago

Is it possible that your plugin does not support urls formatted with the s3:// protocol? As a user I would find this a bit surprising, since s3:// is the standard protocol string for accessing stuff on s3, and the plugin says it supports s3... I tried again with the http:// url for that same resource (http://janelia-cosem-datasets.s3.amazonaws.com/jrc_hela-2/jrc_hela-2.zarr/em/fibsem-uint16/), and got the following error content:

traceback ``` (Fiji Is Just) ImageJ 2.9.0/1.53t; Java 1.8.0_322 [64-bit]; Mac OS X 10.16; 83MB of 2761MB (3%) java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:308) at net.imagej.legacy.DefaultLegacyHooks.interceptRunPlugIn(DefaultLegacyHooks.java:166) at ij.IJ.runPlugIn(IJ.java) at ij.Executer.runCommand(Executer.java:152) at ij.Executer.run(Executer.java:70) at java.lang.Thread.run(Thread.java:750) Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:304) ... 5 more Caused by: java.lang.RuntimeException: Module threw exception at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127) at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63) at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 8XBC1VWCWY2B7RYR; S3 Extended Request ID: 4KRw2sZ8DdH0x5Bp30e6vhDOWnbVCGZOc2jMwT8JK0J7gOWyL375pwv6aJYzGDC1JFmArVSZrM8=; Proxy: null), S3 Extended Request ID: 4KRw2sZ8DdH0x5Bp30e6vhDOWnbVCGZOc2jMwT8JK0J7gOWyL375pwv6aJYzGDC1JFmArVSZrM8= at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1395) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1371) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008) at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1416) at org.embl.mobie.io.util.S3Utils.getS3Client(S3Utils.java:78) at org.embl.mobie.io.ome.zarr.loaders.N5S3OMEZarrImageLoader$N5S3ZarrReaderCreator.create(N5S3OMEZarrImageLoader.java:93) at org.embl.mobie.io.ome.zarr.loaders.N5S3OMEZarrImageLoader.(N5S3OMEZarrImageLoader.java:60) at org.embl.mobie.io.ome.zarr.openers.OMEZarrS3Opener.readKey(OMEZarrS3Opener.java:70) at org.embl.mobie.io.ome.zarr.openers.OMEZarrS3Opener.readURL(OMEZarrS3Opener.java:53) at org.embl.mobie.command.OpenOMEZARRFromS3Command.openAndShow(OpenOMEZARRFromS3Command.java:53) at org.embl.mobie.command.OpenOMEZARRFromS3Command.run(OpenOMEZARRFromS3Command.java:62) at org.scijava.command.CommandModule.run(CommandModule.java:196) at org.scijava.module.ModuleRunner.run(ModuleRunner.java:163) at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124) ... 6 more ```

For reference, neuroglancer handles both the s3 and http versions of the url correctly.

tischi commented 1 year ago

Ok, thanks for letting us know! We will look into it! I am not sure when I will find the time (our main developer of this code unfortunately left us)... Please ping me again if it is urgent for you..