XML generation fails on KLB import

[Xqua]

Hello,

When trying to open a KLB file, I am presented with the KLB import window, however, once the KLB file is selected, saving the dataset as a XML file, or Opening it in BDV fails with this error:

Exception in thread "AWT-EventQueue-0" java.lang.NoClassDefFoundError: bdv/img/cache/VolatileImgCells$CellCache
    at org.janelia.simview.klb.bdv.KlbImgLoader.<init>(KlbImgLoader.java:66)
    at org.janelia.simview.klb.bdv.KlbSpimDataAdapter.createDataset(KlbSpimDataAdapter.java:111)
    at org.janelia.simview.klb.bdv.KlbSpimDataAdapter.writeXML(KlbSpimDataAdapter.java:155)
    at org.janelia.simview.klb.bdv.ui.ViewSetupsConfigDialog.saveXML(ViewSetupsConfigDialog.java:91)
    at org.janelia.simview.klb.bdv.ui.ViewSetupsConfigDialog.actionPerformed(ViewSetupsConfigDialog.java:111)
    at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
    at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
    at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
    at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
    at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
    at java.awt.Component.processMouseEvent(Component.java:6533)
    at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
    at java.awt.Component.processEvent(Component.java:6298)
    at java.awt.Container.processEvent(Container.java:2237)
    at java.awt.Component.dispatchEventImpl(Component.java:4889)
    at java.awt.Container.dispatchEventImpl(Container.java:2295)
    at java.awt.Component.dispatchEvent(Component.java:4711)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4889)
    at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4526)
    at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4467)
    at java.awt.Container.dispatchEventImpl(Container.java:2281)
    at java.awt.Window.dispatchEventImpl(Window.java:2746)
    at java.awt.Component.dispatchEvent(Component.java:4711)
    at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
    at java.awt.EventQueue.access$500(EventQueue.java:97)
    at java.awt.EventQueue$3.run(EventQueue.java:709)
    at java.awt.EventQueue$3.run(EventQueue.java:703)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:90)
    at java.awt.EventQueue$4.run(EventQueue.java:731)
    at java.awt.EventQueue$4.run(EventQueue.java:729)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Caused by: java.lang.ClassNotFoundException: bdv.img.cache.VolatileImgCells$CellCache
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 41 more

[axtimwalde]

That means that the KLB distribution is not built against the bigdataviewer-core artifact that is used in your distribution. I suggest to file this issue in KLB where this can be fixed easily. I am happy to help with this.

[tpietzsch]

I'm also happy to help. @bhoeckendorf

[bhoeckendorf]

I actually have a fix. The real problem is that I still can't upload things to the Fiji updater. I'll try again. If it does not work, I'll upload the fixed .jar somewhere else.

[bhoeckendorf]

Thanks for offering your help. :)

[Xqua]

Ok, thanks ! I made sure to have the latest updates for all packages, but I guess there might be a glitch in who built when ! @bhoeckendorf do you need anything from me ?

[bhoeckendorf]

Thanks for offering @Xqua. For some time now I've not been able to upload anything to the Fiji updater. I've tried from different machines, OSes and networks but haven't had any success. Meanwhile the BigDataViewer APIs changed a little bit and the KLB backend on the updater is now outdated and incompatible.

I'm in the middle of something else currently, but I'll give it another shot later today. If the problem persists I will open a new update site. I'll report back here.

[imagejan]

@bhoeckendorf wrote:

If the problem persists I will open a new update site.

Please do not open a new update site to replace an existing one. Subscribers of the current one will not get notified and end up with obsolete versions of your plugins.

Let's instead try to fix uploading to http://sites.imagej.net/SiMView/ for you if required, possibly with the help of @ctrueden.

[ctrueden]

@imagejan I'm sorry I've been extremely slow in helping @bhoeckendorf with this problem, which is why he's resorting to such extreme measures. We have been (slowly) going back and forth in private email for many months.

The current situation is that I reset the SiMView user's password to a known-to-work password, and tested it myself from my computer, and it worked. But when @bhoeckendorf tries it from any of his machines, on any OSes and at various different network locations, it never works for him.

I guess there must be something different between our setups—in particular, something that @bhoeckendorf always does on his machines that results in the issue? Or perhaps some common misconfiguration of the Manage Update Sites dialog?

@bhoeckendorf What exactly are you specifying for the various fields? Are you specifying webdav:SiMView for Host and ./ for "Directory on Host"? Have you tried with a fresh and untainted installation of Fiji? Another thing we can try is adding another ImageJ wiki account to the list of allowed uploaders for SiMView, and see whether that account is able to upload. Do you have any such wiki accounts I could add for testing?

[Xqua]

Thanks for you guys work on this !

In the mean time, is there a way to maybe downgrade BDV to the KDB dist ? Or better yet, I'm happy to build the KDB against the BDV, but I'd need to know which versions to use ^^ !

[Xqua]

@bhoeckendorf maybe you could release the JAR file here for now until the update site issues are fixed ?

[ctrueden]

I am also happy to upload the JAR from my machine, if @bhoeckendorf shares it (emails to me as an attachment, or makes a cloud share link). I tried to build it myself, but the bhoeckendorf/klb-bdv master branch is not updated with the needed fixes. @bhoeckendorf Could you please push your fixed version?

[ctrueden]

I pushed my (quick) attempt to fix it here: https://github.com/bhoeckendorf/klb-bdv/compare/master...ctrueden:bdv-fixes

[bhoeckendorf]

I did not have any luck uploading the files. This is on a machine that was reinstalled from scratch a few weeks ago. Fiji is a fresh download (no-JRE variant). I configured the upload settings for the update site by deleting the default entry and reintroducing it via the "Add my site" function. It shows the webdav:SiMView entry es expected, although the path was blank. I've tried that and also tried setting the path to ./. I've also reset the password again, just to be certain I'm using the correct one.

@ctrueden I'm going to send you a link to the files. Please apply them. Feel free to reset the password again if needed. No worries for being slow, I have also not exactly been the fastest or the most persistent.

@imagejan I fully understand the problems with opening a new update site, and I am also reluctant. I simply thought it would be the easiest to get the updates out. At the same time, whether or not I'd actually be able to push updates to the new site may help diagnosing the problem.

[Xqua]

@ctrueden There is some dependency error with KLB 3.4

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Keller Lab Block file type (.klb) - BigDataViewer backend 0.2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[WARNING] The POM for org.janelia.simview:klb:jar:0.3.4 is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.092 s
[INFO] Finished at: 2018-02-07T16:24:38-05:00
[INFO] Final Memory: 17M/481M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project klb-bdv: Could not resolve dependencies for project org.janelia.simview:klb-bdv:jar:0.2.0-SNAPSHOT: Failure to find org.janelia.simview:klb:jar:0.3.4 in http://maven.imagej.net/content/groups/public was cached in the local repository, resolution will not be reattempted until the update interval of imagej.public has elapsed or updates are forced -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

[bhoeckendorf]

@Xqua I've send the files you need to @ctrueden so he can apply the updates. You will not currently be able to build it yourself from source without modifying the code. Sorry for not having pushed the code changes to the repos. I'll do so once I get a chance.

[bhoeckendorf]

In the hopes of providing a fast solution, here are the two files: https://www.dropbox.com/s/u6k0cj3hze1w4s9/klb-0.3.6-SNAPSHOT.jar?dl=0 https://www.dropbox.com/s/ufyg9cg3qx2kybn/klb-bdv-0.2.1-SNAPSHOT.jar?dl=0

They need to be placed in the jars subfolder of Fiji. After copying them there, you should have 2 separate versions of klb-[version].jar and klb-bdv-[version].jar in that folder. You should delete the older one.

[Xqua]

thanks @bhoeckendorf ahaha I was fighting against the machine for the last 2 hours ^^ you saved me a night of hair pulling ;)

EDIT: Huston we have signal !

Thanks !!!!!

[bhoeckendorf]

Great, glad it works for you now, @Xqua. Just keep in mind that you now have 'locally modified' files in your Fiji installation. This means that the updater does not handle them automatically anymore. Instead, you can enter 'Advanced mode' (by pressing the button with that name) where you'll be presented with a drop-down menu where you can explicitly choose what to do about the locally modified files. Don't update them for now, because this would actually replace them with the obsolete versions you've been struggling with. You should probably monitor the discussion here so you'll know when we have found a solution for the updater problem. Once this is solved, you should tell Fiji to update the files again. If you don't, they will stay flagged as locally modified and the updater won't deliver future updates of these files.

[Xqua]

@bhoeckendorf yup ! Thanks for the reminder not to update thoose ! I've already set the Watch on this thread ;) Don't hesitate to ping me if you need someone to test once it's pushed !

[ctrueden]

I uploaded the JARs to the SiMView update site. To avoid future pain, we could consider adding Travis CI instructions to the bhoeckendorf/klb-bdv repo that auto-upload successful build artifacts to the SiMView update site.

[bhoeckendorf]

Thanks a lot, @ctrueden. CI is a good idea, I'll look into that. I may need some help on this though, since I'm not sure how to interact with the updater in this context. Furthermore, klb-bdv depends on klb, which includes a C/C++ library that needs to be compiled (for multiple platforms) and bundled into the .jar.

[ctrueden]

@bhoeckendorf I suggest you deploy the klb JAR file(s) to the ImageJ Maven repository at the latest release version. Then depend on that in downstream code, so that your builds are reproducible.

I may need some help on this though, since I'm not sure how to interact with the updater in this context.

Here is how the SciView project does it. This uses the command line updater. It is a little tricky to build the Fiji installation locally before doing the upload-complete-site call (see upload-site-simple.sh), but I'm happy to help as needed. (And your requirements will be far less complex than SciView's.)

[Xqua]

You can also save yourself some time by creating a Docker Image with all the necessary components, and use it as a base for compilation. Can save a looot of time on the build part!