please find a draft solution for having both the authentication and the region passing through N5Factory.
I added a cmd line option to explicitly request the authentication from the backend (at the moment s3 only). This is necessary for the output bucket writing, because the credentials are based on the assumed role in the EMR-serverless session.
Unless the path-like s3 uri (https://s3-us-west-2.amazonaws.com/bucket/...) is used, N5Factory falls back to us-east-1 which does not work for us. Updated ParsedBucket to tolerate path-like uri-s. Maybe we don't really want to use path-like URI-s in the long term but using N5Factory.s3Region to set region obtained from the DefaultAwsRegionProviderChain. See related issue here.
Also note that while this branch works with n5-aws-s3 4.1.1, it fails with key error in 4.1.3 :
mpicbg.spim.data.SpimDataIOException: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: B6CKA6GRDWMBAXWE; S3 Extended Request ID: KUW8//nsXQhv7NIQlZ1pCe801MBlpYnnrwsCtV01VPgy6tCjR+PryJUtmzxJ/3xAA114oWUI4m4=; Proxy: null), S3 Extended Request ID: KUW8//nsXQhv7NIQlZ1pCe801MBlpYnnrwsCtV01VPgy6tCjR+PryJUtmzxJ/3xAA114oWUI4m4=
at net.preibisch.bigstitcher.spark.util.Spark.getSparkJobSpimData2(Spark.java:298)
at net.preibisch.bigstitcher.spark.abstractcmdline.AbstractBasic.loadSpimData2(AbstractBasic.java:30)
at net.preibisch.bigstitcher.spark.SparkAffineFusion.call(SparkAffineFusion.java:130)
Hi Stephan,
please find a draft solution for having both the authentication and the region passing through N5Factory.
I added a cmd line option to explicitly request the authentication from the backend (at the moment s3 only). This is necessary for the output bucket writing, because the credentials are based on the assumed role in the EMR-serverless session.
Unless the path-like s3 uri (
https://s3-us-west-2.amazonaws.com/bucket/...
) is used, N5Factory falls back tous-east-1
which does not work for us. UpdatedParsedBucket
to tolerate path-like uri-s. Maybe we don't really want to use path-like URI-s in the long term but using N5Factory.s3Region to set region obtained from theDefaultAwsRegionProviderChain
. See related issue here.Also note that while this branch works with n5-aws-s3 4.1.1, it fails with key error in 4.1.3 :