search

saalfeldlab / stitching-spark

Reconstruct big images from overlapping tiled images on a Spark cluster.
GNU General Public License v2.0
34 stars 9 forks source link

Stitching and export steps fail on AWS #27

Closed igorpisarev closed 4 years ago

igorpisarev commented 4 years ago

The following error occurs when trying to run stitching and export steps when data is stored in AWS S3:

Exception in thread "main" com.esotericsoftware.kryo.KryoException: java.lang.ClassCastException: [C cannot be cast to [B
Serialization trace:
sessionId (java.lang.String)
key (sun.security.util.MemoryCache$SoftCacheEntry)
cacheMap (sun.security.util.MemoryCache)
sessionHostPortCache (sun.security.ssl.SSLSessionContextImpl)
clientCache (sun.security.ssl.SSLContextImpl$TLSContext)
context (sun.security.ssl.SSLSocketFactoryImpl)
socketfactory (com.amazonaws.http.conn.ssl.SdkTLSSocketFactory)
map (org.apache.http.config.Registry)
socketFactoryRegistry (org.apache.http.impl.conn.DefaultHttpClientConnectionOperator)
connectionOperator (org.apache.http.impl.conn.PoolingHttpClientConnectionManager)
cm (com.amazonaws.http.apache.client.impl.SdkHttpClient)
httpClient (com.amazonaws.http.AmazonHttpClient)
client (com.amazonaws.services.s3.AmazonS3Client)
client (com.amazonaws.util.ServiceClientHolderInputStream)
in (com.amazonaws.services.s3.AmazonS3Client$2)
in (com.amazonaws.services.s3.internal.DigestValidationInputStream)
in (com.amazonaws.services.s3.model.S3ObjectInputStream)
inputStream (ij.io.FileInfo)
fileInfo (ij.ImagePlus)
imp (net.imglib2.img.imageplus.FloatImagePlus)
slices (net.imglib2.view.StackView)
source (net.imglib2.view.ExtendedRandomAccessibleInterval)
sourceA (net.imglib2.view.RandomAccessiblePairNullable)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:82)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:577)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:68)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
    at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:95)
    at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:21)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:577)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:68)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
    at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:95)
    at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:21)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:348)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:289)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:82)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:22)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
    at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:241)
    at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$blockifyObject$2.apply(TorrentBroadcast.scala:291)
    at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$blockifyObject$2.apply(TorrentBroadcast.scala:291)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1380)
    at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:292)
    at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:127)
    at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:88)
    at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
    at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
    at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1482)
    at org.apache.spark.api.java.JavaSparkContext.broadcast(JavaSparkContext.scala:650)
    at org.janelia.stitching.PipelineStitchingStepExecutor.computePairwiseShifts(PipelineStitchingStepExecutor.java:398)
    at org.janelia.stitching.PipelineStitchingStepExecutor.preparePairwiseShiftsMulti(PipelineStitchingStepExecutor.java:346)
    at org.janelia.stitching.PipelineStitchingStepExecutor.run(PipelineStitchingStepExecutor.java:108)
    at org.janelia.stitching.StitchingSpark.run(StitchingSpark.java:87)
    at org.janelia.stitching.StitchingSpark.main(StitchingSpark.java:34)
Caused by: java.lang.ClassCastException: [C cannot be cast to [B
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:25)
    at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:577)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:68)
    ... 96 more
igorpisarev commented 4 years ago

This happens when the flatfield images are broadcasted to all Spark executors. Even though it works fine when the flatfield images are loaded from the filesystem, it breaks somewhere in Kryo internals when trying to serialize the images loaded from S3. Some debugging showed that ImagePlus has an inner field fileInfo where it stores a reference to the underlying InputStream. For S3 there is a lot more going on in the stream object hierarchy than for the filesystem, which causes errors in Kryo serialization.

A reasonable solution would be to create a copy of the flatfield images, so that there are no hidden object references stored internally.