search

scalableminds / webknossos

Visualize, share and annotate your large 3D images online
https://webknossos.org
GNU Affero General Public License v3.0
124 stars 23 forks source link

upload of aligned tif series #917

Closed deployboy closed 8 years ago

deployboy commented 8 years ago

Hi,

I just tried to upload a series of aligned tifs (size=180MB) with the "Upload Dataset" function. However, after the upload is done I get the error "Folder is locked by another lock keeper." Could you have a look into it please ? I've also uploaded the folder on the wk server manually, the folder is called "tt". Thanks, Best, Philip

Reported by: Philip Laserstein (lasersteinp@neuro.mpg.de) on master

Log Time

tmbo commented 8 years ago

The uplaod failes due to an exception. I will have a look into that tomorrow:

ava.lang.ClassCastException: java.awt.image.DataBufferUShort cannot be cast to java.awt.image.DataBufferByte
    at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$$anonfun$toRawImage$1.apply(ImageDataSourceType.scala:237) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$$anonfun$toRawImage$1.apply(ImageDataSourceType.scala:229) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at scala.Option.flatMap(Option.scala:170) ~[org.scala-lang.scala-library-2.10.3.jar:na]
    at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$class.toRawImage(ImageDataSourceType.scala:228) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.TiffDataSourceType$.toRawImage(ImageDataSourceType.scala:23) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$class.extractImageInfo(ImageDataSourceType.scala:121) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.TiffDataSourceType$.extractImageInfo(ImageDataSourceType.scala:23) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$$anonfun$extractLayers$2.apply(ImageDataSourceType.scala:157) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$$anonfun$extractLayers$2.apply(ImageDataSourceType.scala:154) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) ~[org.scala-lang.scala-library-2.10.3.jar:na]
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) ~[org.scala-lang.scala-library-2.10.3.jar:na]
    at scala.collection.immutable.Map$Map1.foreach(Map.scala:109) ~[org.scala-lang.scala-library-2.10.3.jar:na]
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) ~[org.scala-lang.scala-library-2.10.3.jar:na]
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) ~[org.scala-lang.scala-library-2.10.3.jar:na]
    at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$class.extractLayers(ImageDataSourceType.scala:154) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.TiffDataSourceType$.extractLayers(ImageDataSourceType.scala:23) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$class.convertToKnossosStructure(ImageDataSourceType.scala:178) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.TiffDataSourceType$.convertToKnossosStructure(ImageDataSourceType.scala:23) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$class.importDataSource(ImageDataSourceType.scala:99) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.TiffDataSourceType$.importDataSource(ImageDataSourceType.scala:23) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.DataSourceInboxHelper$$anonfun$transformToDataSource$2.apply(DataSourceInboxHelper.scala:51) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.DataSourceInboxHelper$$anonfun$transformToDataSource$2.apply(DataSourceInboxHelper.scala:47) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.LockKeeperHelper$$anonfun$withLock$1.apply(LockKeeper.scala:22) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.braingames.binary.repository.LockKeeperHelper$$anonfun$withLock$1.apply(LockKeeper.scala:21) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.util.tools.Fox$$anonfun$flatMap$1.apply(Fox.scala:85) ~[com.scalableminds.util_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at com.scalableminds.util.tools.Fox$$anonfun$flatMap$1.apply(Fox.scala:83) ~[com.scalableminds.util_2.10-6.10.9-master-fix.jar:6.10.9-master-fix]
    at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251) ~[org.scala-lang.scala-library-2.10.3.jar:na]
    at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:249) ~[org.scala-lang.scala-library-2.10.3.jar:na]
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) [org.scala-lang.scala-library-2.10.3.jar:na]
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1]
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1]
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1]
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1]
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) [org.scala-lang.scala-library-2.10.3.jar:na]
    at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1]
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1]
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1]
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [org.scala-lang.scala-library-2.10.3.jar:na]
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [org.scala-lang.scala-library-2.10.3.jar:na]
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [org.scala-lang.scala-library-2.10.3.jar:na]
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [org.scala-lang.scala-library-2.10.3.jar:na]
jfrohnhofen commented 8 years ago

I have seen this exception before and this usually means, that the images are 16bit tiff files, which webKnossos can not properly convert to 8bit images.

Johannes Frohnhofen

johannes.frohnhofen@scalableminds.com mailto:johannes.frohnhofen@scalableminds.com http://scm.io http://scm.io/

scalableminds UG (haftungsbeschränkt) & Co. KG Amtsgericht Potsdam, HRA 5753 Geschäftsführer: Tom Bocklisch, Tom Herold, Norman Rzepka, Thomas Werkmeister

Am 05.11.2015 um 10:56 schrieb Tom Bocklisch notifications@github.com:

The uplaod failes due to an exception. I will have a look into that tomorrow:

ava.lang.ClassCastException: java.awt.image.DataBufferUShort cannot be cast to java.awt.image.DataBufferByte at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$$anonfun$toRawImage$1.apply(ImageDataSourceType.scala:237) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$$anonfun$toRawImage$1.apply(ImageDataSourceType.scala:229) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at scala.Option.flatMap(Option.scala:170) ~[org.scala-lang.scala-library-2.10.3.jar:na] at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$class.toRawImage(ImageDataSourceType.scala:228) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.TiffDataSourceType$.toRawImage(ImageDataSourceType.scala:23) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$class.extractImageInfo(ImageDataSourceType.scala:121) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.TiffDataSourceType$.extractImageInfo(ImageDataSourceType.scala:23) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$$anonfun$extractLayers$2.apply(ImageDataSourceType.scala:157) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$$anonfun$extractLayers$2.apply(ImageDataSourceType.scala:154) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) ~[org.scala-lang.scala-library-2.10.3.jar:na] at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) ~[org.scala-lang.scala-library-2.10.3.jar:na] at scala.collection.immutable.Map$Map1.foreach(Map.scala:109) ~[org.scala-lang.scala-library-2.10.3.jar:na] at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) ~[org.scala-lang.scala-library-2.10.3.jar:na] at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) ~[org.scala-lang.scala-library-2.10.3.jar:na] at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$class.extractLayers(ImageDataSourceType.scala:154) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.TiffDataSourceType$.extractLayers(ImageDataSourceType.scala:23) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$class.convertToKnossosStructure(ImageDataSourceType.scala:178) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.TiffDataSourceType$.convertToKnossosStructure(ImageDataSourceType.scala:23) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.ImageDataSourceTypeHandler$class.importDataSource(ImageDataSourceType.scala:99) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.TiffDataSourceType$.importDataSource(ImageDataSourceType.scala:23) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.DataSourceInboxHelper$$anonfun$transformToDataSource$2.apply(DataSourceInboxHelper.scala:51) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.DataSourceInboxHelper$$anonfun$transformToDataSource$2.apply(DataSourceInboxHelper.scala:47) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.LockKeeperHelper$$anonfun$withLock$1.apply(LockKeeper.scala:22) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.braingames.binary.repository.LockKeeperHelper$$anonfun$withLock$1.apply(LockKeeper.scala:21) ~[com.scalableminds.braingames-binary_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.util.tools.Fox$$anonfun$flatMap$1.apply(Fox.scala:85) ~[com.scalableminds.util_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at com.scalableminds.util.tools.Fox$$anonfun$flatMap$1.apply(Fox.scala:83) ~[com.scalableminds.util_2.10-6.10.9-master-fix.jar:6.10.9-master-fix] at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251) ~[org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:249) ~[org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) [org.scala-lang.scala-library-2.10.3.jar:na] at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1] at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1] at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1] at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1] at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) [org.scala-lang.scala-library-2.10.3.jar:na] at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) [com.typesafe.akka.akka-actor_2.10-2.2.1.jar:2.2.1] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [org.scala-lang.scala-library-2.10.3.jar:na] — Reply to this email directly or view it on GitHub https://github.com/scalableminds/oxalis/issues/917#issuecomment-154011449.

boergens commented 8 years ago

Would it be hard to make it possible to upload 16bit tiffs?

jfrohnhofen commented 8 years ago

Since webKnossos (and most displays) can only show 8bit grayscale data, we need to convert the data at some point. We can certainly do that as part of the import. However, there are two issues:

1) the upload and import will take twice as long without any benefit

2) how should the conversion take place? i haven’t looked at the images this time yet, but last time this problem occurred, the images only used a tiny fraction of the 16bit spectrum. if we then map the full 16bit range to 8bit, the resulting images will be mostly black. so the user either needs to define the range of used colors or we need to determine it dynamically, which would require the server to read the dataset twice, slowing down import even more.

So in summary, possible, but not ideal.

Best Johannes

Johannes Frohnhofen

johannes.frohnhofen@scalableminds.com mailto:johannes.frohnhofen@scalableminds.com http://scm.io http://scm.io/

scalableminds UG (haftungsbeschränkt) & Co. KG Amtsgericht Potsdam, HRA 5753 Geschäftsführer: Tom Bocklisch, Tom Herold, Norman Rzepka, Thomas Werkmeister

Am 05.11.2015 um 11:03 schrieb Kevin Michael Boergens notifications@github.com:

Would it be hard to make it possible to upload 16bit tiffs?

— Reply to this email directly or view it on GitHub https://github.com/scalableminds/oxalis/issues/917#issuecomment-154013924.

boergens commented 8 years ago

@jfrohnhofen I understand Unfortunately, I still think we need it. There are too many small 16 bit datasets out there It should only be activated for HTML upload and it should be dynamically ranged.