carandraug
commented
8 years ago Making use of imglib2 and imagej ops fixes it.
Closed carandraug closed 8 years ago
carandraug
commented
8 years ago Making use of imglib2 and imagej ops fixes it.
graemeball
commented
8 years ago Edit: in my original comment below I did not pay attention to the fact you are trying to run the jython script --headless, so you can probably disregard most of it apart from the last sentence. I get the same exception you do when I try to run headless. I suspect you will find many (most?) parts of SIMcheck do not run headless unfortunately, so unless you want to re-implement the bulk of it (SIMcheck2??), it may be easier to pursue the Xvfb option described here: Headless - Fiji
David: I tried the jython script below & it didn't give me a NPE... did you figure out what the problem was with your script?
from ij import IJ
from loci.plugins import BF
from SIMcheck import Util_SItoPseudoWidefield
from SIMcheck import I1l
# this example image is a stack
fpath = "/Users/gball/Documents/TestData/SIM/NR-beads-200nm-1511-SI_30pct.dv"
imp = BF.openImagePlus(fpath)[0]
proj_mode = Util_SItoPseudoWidefield.ProjMode.AVG
norm_imp = I1l.normalizeImp(imp)
impPWF = Util_SItoPseudoWidefield().exec(norm_imp, 5, 3, proj_mode) # throws NullPointerException
impPWF.show()
# this example image is a hyperstack
fpath2 = "/Users/gball/Documents/InTray/SIMcheck/Test/LamB_cell1_TL.tif"
imp2 = BF.openImagePlus(fpath2)[0]
proj_mode = Util_SItoPseudoWidefield.ProjMode.AVG
norm_imp2 = I1l.normalizeImp(imp2)
impPWF2 = Util_SItoPseudoWidefield().exec(norm_imp2, 5, 3, proj_mode) # throws NullPointerException
impPWF2.show()and why did you need to use ImageJ2 ops & SciJava? (did you conclude the Scaler plugin was borken?)
I was trying to avoid adding that stuff as a dependency of the core plugins, because it will only work in Fiji or ImageJ2 (if some subset of the plugin options needs ImgLib2 functionality that seems like less of a problem -- e.g. I was planning to add 3D FFT to the Fourier checks, but fall back to the 2D version where not available).
carandraug
commented
8 years ago @graemeball wrote:
David: I tried the jython script below & it didn't give me a NPE... did you figure out what the problem was with your script?
from ij import IJ from loci.plugins import BF from SIMcheck import Util_SItoPseudoWidefield from SIMcheck import I1l # this example image is a stack fpath = "/Users/gball/Documents/TestData/SIM/NR-beads-200nm-1511-SI_30pct.dv" imp = BF.openImagePlus(fpath)[0] proj_mode = Util_SItoPseudoWidefield.ProjMode.AVG norm_imp = I1l.normalizeImp(imp) impPWF = Util_SItoPseudoWidefield().exec(norm_imp, 5, 3, proj_mode) # throws NullPointerException impPWF.show()[...]
That doesn't work for me. Are you really running it on headless mode?
$ cat foo.py
from ij import IJ
from loci.plugins import BF
from SIMcheck import Util_SItoPseudoWidefield
from SIMcheck import I1l
# this example image is a stack
fpath = "mustafa.aydogan/CEP97_20160120_b_10.dv"
imp = BF.openImagePlus(fpath)[0]
proj_mode = Util_SItoPseudoWidefield.ProjMode.AVG
norm_imp = I1l.normalizeImp(imp)
impPWF = Util_SItoPseudoWidefield().exec(norm_imp, 5, 3, proj_mode) # throws NullPointerException
impPWF.show()
$ .opt/Fiji.app/ImageJ-linux64 --headless foo.py
[... trim a lot of ImageJ output]
java.lang.NullPointerException
at ij.plugin.Scaler.showDialog(Scaler.java:207)
at ij.plugin.Scaler.run(Scaler.java:42)
at ij.IJ.runPlugIn(IJ.java:182)
[...]and why did you need to use ImageJ2 ops & SciJava? (did you conclude the Scaler plugin was borken?) I was trying to avoid adding that stuff as a dependency of the core plugins, because it will only work in Fiji or ImageJ2 (if some subset of the plugin options needs ImgLib2 functionality that seems like less of a problem -- e.g. I was planning to add 3D FFT to the Fourier checks, but fall back to the 2D version where not available).
Yes, I think it is no good if you run in headless mode. I also tried to call directly the Scaler plugin instead of IJ.run() but I couldn't figure it out.
graemeball
commented
8 years ago haha, simultaneous comments, see above ;-)
I seem to remember I couldn't get the Scaler plugin to work calling it programmatically as you guessed -- it was driving me nuts, and I was in a hurry at the time -- hence the nasty hacky implementation you found.
carandraug
commented
8 years ago [...] I get the same exception you do when I try to run headless. I suspect you will find many (most?) parts of SIMcheck do not run headless unfortunately, so unless you want to re-implement the bulk of it (SIMcheck2??), [...]
Indeed, I was hoping to use other bits of SIMcheck in headless mode and found them no good. I know you don't really like imglib2 and that you're also trying to keep SIMcheck running in imagej1 so I was expecting it.
Porting SIMcheck as a imglib2 plugin would be too much work so if you don't want this type of changes I'll just reimplement the bits I need in python. The comments say that the MCNR code was a port from Rainer's Matlab code. Do you still have the original code then?
graemeball
commented
8 years ago Indeed, I was hoping to use other bits of SIMcheck in headless mode and found them no good. I know you don't really like imglib2 and that you're also trying to keep SIMcheck running in imagej1 so I was expecting it.
I'm not really against ImgLib2, and it is supposed to be somewhat stable now, so I would like to give that a try. The documentation seems pretty thin though, which is frustrating. The ImageJ2 / SciJava / Ops / Scifio stuff on the other hand: is it supposed to be stable-ish now? It seemed very buggy & prone to change in the recent past. Are you in touch with those guys much via chat etc.?
If everyone using SIMcheck is using Fiji, I guess it does not matter about ImageJ1 compatibility. We should probably ask around. We could always offer people the option of downloading the published 1.0 version while continuing to update a newer version that is not necessarily ImageJ1 compatible. Does that sound reasonable?
Did you try the Xvfb option by the way? I know it's not very elegant, and you're hosed if a pop-up asks for input, but it may be the most pragmatic option in the short term.
Porting SIMcheck as a imglib2 plugin would be too much work so if you don't want this type of changes I'll just reimplement the bits I need in python. The comments say that the MCNR code was a port from Rainer's Matlab code. Do you still have the original code then?
If you want to re-implement the modulation contrast bit in python, you'll have to call a fast 1D discrete Fourier transform from the python rather than implementing the DFT in python or it'll be really slow. I had to make some effort to speed that up even in Java by caching Fourier coefficients & multi-threading it. Actually though, that plugin may work headless - did you try it?
P.S. No, I don't think I do have Rainer's original MATLAB code -- must've gotten lost in copying stuff between laptops when I moved -- sorry about that. There is a description of Rainer's method versus mine in the big javadoc comment at the top of the in the Raw_ModContrast class, but Rainer's method gave a very different result and required the user to define a background region.
tlambert03
commented
8 years ago been watching this exchange... I've also run into issues in the past trying to run SIMcheck headless from the command line and ended up reimplementing the individual modules I wanted in python. So far I have simple ones like RIH and TIV, but I ran into trouble with MCNR and never got back to it. If either of you do end up making little python bits for various modules (particularly MCNR), I'd be very interested... or if you want any of the ones that I've already generated, I'm happy to share (they're not all that polished, but they do "work").
graemeball
commented
8 years ago I just checked for the case of Raw_ModContrast, and was disappointed that it also chokes on --headless
However, it's quite easy to work around in that case -- the problem is the Calibration Bar... command, which I confirmed by commenting this out & recompiling. The plugin then runs fine --headless using a script like this:
from ij import IJ
from loci.plugins import BF
from SIMcheck import Raw_ModContrast
from ij.io import FileSaver
from java.awt import GraphicsEnvironment
# we can check if we are running headless -- a plugin could do the same...
if GraphicsEnvironment.isHeadless():
IJ.log("Running headless!")
# hard-coded input & output
fpath = "/Users/gball/Documents/TestData/SIM/NR-beads-200nm-1511-SI_30pct.dv"
fpath_out = "/Users/gball/Desktop/NR-beads-200nm-1511-SI_30pct_MCN.tif"
imp = BF.openImagePlus(fpath)[0]
results = Raw_ModContrast().exec(imp) # the exec method returns a ResultSet container object
impMCNR = results.getImp(0)
FileSaver(impMCNR).saveAsTiffStack(fpath_out)You may have spotted in the snippet above that I also test whether we are running headless. It would be a simple matter to add a check like this to some plugins in order to skip or replace sections in a headless environment.
tlambert03
commented
8 years ago Forgive me if this is a tangent... but I'm having difficulties running that code from command line (OS X) in general... headless or not (it works fine when run within fiji). From command line, it begins opening the file and then hangs showing "Populating OME metadata" (but doesn't report any errors). I originally thought that Bioformats had issues in general running headless. But you don't seem to be having that issue... am I missing something here? are you using Xvfb?
graemeball
commented
8 years ago Forgive me if this is a tangent... but I'm having difficulties running that code from command line (OS X) in general... headless or not (it works fine when run within fiji). From command line, it begins opening the file and then hangs showing "Populating OME metadata" (but doesn't report any errors). I originally thought that Bioformats had issues in general running headless. But you don't seem to be having that issue... am I missing something here? are you using Xvfb?
I'm calling the scripts like this from OSX command line (10.9.5, Mavericks; via iTerm2, but I don't think that matters):
/Applications/Fiji.app/Contents/MacOS/ImageJ-macosx --headless ~/Desktop/run_mcnr.py
and no, I don't think I'm running Xvfb -- looks like that involves recompiling Xquartz with --enable-xvfb. Probably easier & more useful on linux, so I'll give that a go when I get a chance.
To my great surprise the Raw_IntensityProfiles plugin seems to run fine headless. No doubt some of the others will have big problems though.
P.S. you're using a Mac these days?!
tlambert03
commented
8 years ago re: using a mac ... you may be confusing me with @carandraug :) (sorry for hijacking the thread) that's how I call the script too ... still hangs when trying to open the file. But it's obviously something specific to my setup if you're not having the same problem, so I'll try to figure it out. thanks!
graemeball
commented
8 years ago @tlambert03 you're right, I didn't spot who posted that message <:-|
I wonder, what bio-formats options have you got set currently? I think it stores the options you were using last, and some of them throw up a dialog, right? (just a thought -- might be something completely different)
graemeball
commented
8 years ago Probably won't get this finished until next week (I'm away for a long weekend, no charger, 18% and falling...) but I've been trying each check in turn to see if/where they have problems headless:
carandraug
commented
8 years ago I'm not really against ImgLib2, and it is supposed to be somewhat stable now, so I would like to give that a try. The documentation seems pretty thin though, which is frustrating. The ImageJ2 / SciJava / Ops / Scifio stuff on the other hand: is it supposed to be stable-ish now? It seemed very buggy & prone to change in the recent past. Are you in touch with those guys much via chat etc.?
If the frequency with each an update breaks ImageJ is anything to go by, I'd say that yes, it's a lot more stable. They're still a bit lacking on documentation but the community is quite helpful.
If everyone using SIMcheck is using Fiji, I guess it does not matter about ImageJ1 compatibility. We should probably ask around. We could always offer people the option of downloading the published 1.0 version while continuing to update a newer version that is not necessarily ImageJ1 compatible. Does that sound reasonable?
Sounds reasonable to me. But eventually you'll implement new stuff on the new version of ImageJ and there may be people who want that ported to imagej1. Still, I think you can always tell them to get fiji with imglib2. Still, I will imagine that this will be a fair amount of work so it's up to you if that's worthy. I thought my needs as wanting to call it headless mode would be odd and not very important for you but seems like @tlambert03 is also having the issue.
Did you try the Xvfb option by the way? I know it's not very elegant, and you're hosed if a pop-up asks for input, but it may be the most pragmatic option in the short term.
I just gave it a very quick try and indeed it got stuck somewhere. I'm sure I can fix it but this is part of a make recipe that I want others (on a Mac) to be able to run. So I don't want a workaround that is mostly for Linux.
but I've been trying each check in turn to see if/where they have problems headless [...]
Should I rename this issue to "SIMcheck not that useful in headless mode"?
graemeball
commented
8 years ago But eventually you'll implement new stuff on the new version of ImageJ and there may be people who want that ported to imagej1. Still, I think you can always tell them to get fiji with imglib2. Still, I will imagine that this will be a fair amount of work so it's up to you if that's worthy. I thought my needs as wanting to call it headless mode would be odd and not very important for you but seems like @tlambert03 is also having the issue.
After thinking about it, personally I'd be happy to tell people to use Fiji for new updates (provided we still give them a link to get the old / published ImageJ1 version). Lothar always favoured Fiji as well unless I am mistaken. I think a large minority of those using SIMcheck will want to batch process their SIM images, perhaps in headless mode, so it would be good to fix all the issues that prevent this. I am not sure it is worth releasing any more ImageJ1-only versions, since some of the new features (3D FFT, headless) will rely on ImgLib2 and/or ImageJ2.
Should I rename this issue to "SIMcheck not that useful in headless mode"?
Should we go ahead and merge your pull request here, and open a more general issue "SIMcheck does not run in headless mode"?
carandraug
commented
8 years ago Should we go ahead and merge your pull request here, and open a more general issue "SIMcheck does not run in headless mode"?
Go ahead then. A deeper fix on this class could be made so it becomes an image-op and integrate better with imglib2 (instead of just replacing the part that does the scaling).
iandobbie
commented
8 years ago Carnë Draug notifications@github.com writes:
Sounds reasonable to me. But eventually you'll implement new stuff on the new version of ImageJ and there may be people who want that ported to imagej1. Still, I think you can always tell them to get fiji with imglib2. Still, I will imagine that this will be a fair amount of work so it's up to you if that's worthy. I thought my needs as wanting to call it headless mode would be odd and not very important for you but seems like @tlambert03 is also having the issue.
I'm thinking about doing some headless SIMcheck stuff to implement some of the SIM OMEROmetrics stuff we have been thinking about.
As tot he imglib2, I think it would impact almost no-one if you just say "requires fiji". Maybe we should try it and see if anyone moans? I wonder how many downloads we get from github, compared the fiji download site.
Should I rename this issue to "SIMcheck not that useful in headless mode"?
Or maybe just "broken!"
Ian
I'm writing an ImageJ script to run in headless mode that makes use of SIMcheck (I'm only using the SI to pseudowidefield at the moment but I'm hoping to use it more later) but it fails:
The stack:
So I tried to reimplement the whole thing directly calling ij.plugin.scaler (not sure if that solve it anyway) instead of IJ.run (which is what SIMcheck is doing) but then
projectPandA()(which would be needed) is a private method.