search

tkataras / Automatic-Cell-Counting-with-TWS

A repository for the suite of scripts I use to perform automatic cell quantification.
GNU General Public License v3.0
9 stars 6 forks source link

Error while loading header train #69

Closed sbg30 closed 1 week ago

sbg30 commented 8 months ago

Hi,

Thank you very much for this pluggin that could be a wonderful help for day-to-day experiments !

I am running into an error I am not able to overcome: When trying to run step 1 after completing every steps in the readme file I face : "Error while loading header train" and clicking ok "Could not apply Classifier!"

I trained the models on 8-bit images that were resized to 1024 pxl, idem for Validation images I only had to modify the ACCT_1.ijm script because it was not able to execute python: so every "python" had to be replaced with "python3"

The console displays this:

java.io.StreamCorruptedException: invalid stream header: 00000001 java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:943) java.io.ObjectInputStream.(ObjectInputStream.java:401) trainableSegmentation.WekaSegmentation.internalLoadClassifier(WekaSegmentation.java:655) trainableSegmentation.WekaSegmentation.loadClassifier(WekaSegmentation.java:733) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) bsh.Reflect.invokeMethod(Reflect.java:131) bsh.Reflect.invokeObjectMethod(Reflect.java:77) bsh.Name.invokeMethod(Name.java:852) bsh.BSHMethodInvocation.eval(BSHMethodInvocation.java:69) bsh.BSHPrimaryExpression.eval(BSHPrimaryExpression.java:96) bsh.BSHPrimaryExpression.eval(BSHPrimaryExpression.java:41) bsh.BSHBlock.evalBlock(BSHBlock.java:125) bsh.BSHBlock.eval(BSHBlock.java:75) bsh.BSHBlock.eval(BSHBlock.java:41) bsh.BSHForStatement.eval(BSHForStatement.java:105) bsh.Interpreter.eval(Interpreter.java:659) bsh.Interpreter.eval(Interpreter.java:750) bsh.Interpreter.eval(Interpreter.java:739) org.scijava.plugins.scripting.beanshell.BeanshellScriptEngine.eval(BeanshellScriptEngine.java:68) org.scijava.script.ScriptModule.run(ScriptModule.java:157) org.scijava.module.ModuleRunner.run(ModuleRunner.java:165) org.scijava.module.ModuleRunner.call(ModuleRunner.java:124) org.scijava.module.ModuleRunner.call(ModuleRunner.java:63) org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750)

at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:943)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:401)
at trainableSegmentation.WekaSegmentation.internalLoadClassifier(WekaSegmentation.java:655)
at trainableSegmentation.WekaSegmentation.loadClassifier(WekaSegmentation.java:733)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at bsh.Reflect.invokeMethod(Reflect.java:131)
at bsh.Reflect.invokeObjectMethod(Reflect.java:77)
at bsh.Name.invokeMethod(Name.java:852)
at bsh.BSHMethodInvocation.eval(BSHMethodInvocation.java:69)
at bsh.BSHPrimaryExpression.eval(BSHPrimaryExpression.java:96)
at bsh.BSHPrimaryExpression.eval(BSHPrimaryExpression.java:41)
at bsh.BSHBlock.evalBlock(BSHBlock.java:125)
at bsh.BSHBlock.eval(BSHBlock.java:75)
at bsh.BSHBlock.eval(BSHBlock.java:41)
at bsh.BSHForStatement.eval(BSHForStatement.java:105)
at bsh.Interpreter.eval(Interpreter.java:659)
at bsh.Interpreter.eval(Interpreter.java:750)
at bsh.Interpreter.eval(Interpreter.java:739)
at org.scijava.plugins.scripting.beanshell.BeanshellScriptEngine.eval(BeanshellScriptEngine.java:68)
at org.scijava.script.ScriptModule.run(ScriptModule.java:157)
at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

java.lang.NullPointerException hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750)

at hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

java.lang.NullPointerException hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750)

at hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

java.lang.NullPointerException hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750)

at hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

java.lang.NullPointerException hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750)

at hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

java.lang.NullPointerException hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750)

at hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

java.lang.NullPointerException hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750)

at hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

java.lang.NullPointerException hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750)

at hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

java.lang.NullPointerException hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090) trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750)

at hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7090)
at trainableSegmentation.WekaSegmentation$9.call(WekaSegmentation.java:7062)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

Exception in thread "Thread-42" java.lang.NullPointerException trainableSegmentation.WekaSegmentation.applyClassifier(WekaSegmentation.java:6763) trainableSegmentation.WekaSegmentation$1ApplyClassifierThread.run(WekaSegmentation.java:5508)

at trainableSegmentation.WekaSegmentation.applyClassifier(WekaSegmentation.java:6763)
at trainableSegmentation.WekaSegmentation$1ApplyClassifierThread.run(WekaSegmentation.java:5508)

[ERROR] Sourced file: inline evaluation of: /* * Author: Tyler Jang, Theo * Date: 3/3/2022 * * Input: Classifiers to a . . . '' : Method Invocation segmentator.applyClassifier : at Line: 72 : in file: inline evaluation of:/ Author: Tyler Jang, Theo Date: 3/3/2022 * Input: Classifiers to a . . . '' : segmentator .applyClassifier ( image , tilesPerDim , 0 , true )

Target exception: java.lang.NullPointerException

at bsh.BSHMethodInvocation.eval(BSHMethodInvocation.java:91)
at bsh.BSHPrimaryExpression.eval(BSHPrimaryExpression.java:96)
at bsh.BSHPrimaryExpression.eval(BSHPrimaryExpression.java:41)
at bsh.BSHAssignment.eval(BSHAssignment.java:71)
at bsh.BSHBlock.evalBlock(BSHBlock.java:125)
at bsh.BSHBlock.eval(BSHBlock.java:75)
at bsh.BSHBlock.eval(BSHBlock.java:41)
at bsh.BSHIfStatement.eval(BSHIfStatement.java:42)
at bsh.BSHBlock.evalBlock(BSHBlock.java:125)
at bsh.BSHBlock.eval(BSHBlock.java:75)
at bsh.BSHBlock.eval(BSHBlock.java:41)
at bsh.BSHIfStatement.eval(BSHIfStatement.java:42)
at bsh.BSHBlock.evalBlock(BSHBlock.java:125)
at bsh.BSHBlock.eval(BSHBlock.java:75)
at bsh.BSHBlock.eval(BSHBlock.java:41)
at bsh.BSHForStatement.eval(BSHForStatement.java:105)
at bsh.BSHBlock.evalBlock(BSHBlock.java:125)
at bsh.BSHBlock.eval(BSHBlock.java:75)
at bsh.BSHBlock.eval(BSHBlock.java:41)
at bsh.BSHForStatement.eval(BSHForStatement.java:105)
at bsh.Interpreter.eval(Interpreter.java:659)
at bsh.Interpreter.eval(Interpreter.java:750)
at bsh.Interpreter.eval(Interpreter.java:739)
at org.scijava.plugins.scripting.beanshell.BeanshellScriptEngine.eval(BeanshellScriptEngine.java:68)
at org.scijava.script.ScriptModule.run(ScriptModule.java:157)
at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

Caused by: java.lang.NullPointerException at trainableSegmentation.WekaSegmentation.applyClassifier(WekaSegmentation.java:5566) at trainableSegmentation.WekaSegmentation.applyClassifier(WekaSegmentation.java:6561) at trainableSegmentation.WekaSegmentation.applyClassifier(WekaSegmentation.java:6461) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at bsh.Reflect.invokeMethod(Reflect.java:131) at bsh.Reflect.invokeObjectMethod(Reflect.java:77) at bsh.Name.invokeMethod(Name.java:852) at bsh.BSHMethodInvocation.eval(BSHMethodInvocation.java:69) ... 32 more

Thank you very much for your help

Best

S

TylerJang commented 8 months ago

Hi,

Thanks for the kind words. My initial impressions of the problem lead me to guess that something in the setup was more likely to be the issue. The error that states that it failed to load the header is a problem that implies to me that the created classifiers were not created on the same format of images as the ones being given to it in the training_area/Validation_images folder. For example, I've run into this problem many times where I would create classifiers on images that I converted to 8-bit, but forgot to change the images in training_area/Validation_images to 8-bit as well. Or I would have the inverse problem where the classifiers were trained on non-8-bit images. I would also note that you should check that the images were converted to 8-bit and not 8-bit color.

This also means that your Validation_images should also be in 1024 pixels like you did during training and are the same dimensions (width x height). If not, then this may cause the error as well.

I don't believe that changing the code from "python" to the newer version of python, "python3" would cause this particular issue since the error is happening in ImageJ macro code.

If this doesn't work, could you provide more details about the way you formatted your images so I can try and help?

Thank you, Tyler Jang

sbg30 commented 8 months ago

Thank you very much for getting back. Following your message I started over, to make sure I did not have any discrepancy between images used to train classifiers and validation images

I formatted the images following this:

But after all this, I still encounter the error "Error while loading header train" and clicking when ok: "Could not apply Classifier!"

Is there a way I can troubleshoot this problem differently ?

Thank you very much for your help again

Best

Simon

TylerJang commented 8 months ago

Hello,

Sorry for the delay in my response since I've been out for the holidays. The initial description of your process to format the images sounds fine, unless the images are not being saved correctly in your folders somehow. How are stacking the images? Since the way the sentence is phrased, "for training images, I piles them in a stack as recommended, then adjusted the size with Adjust > Image > Size to 1024 x 768 pixels", implies that you are stacking the images and then adjusting the size. If the images are of different sizes you would expect some pop up window using the Images to Stack command with a message like image.

If this is not popping up for you, it would imply that all your images already are the same size just not 1024x768, then why not just use the images at that size without readjusting the size? The reason I suggest this is that when images are in a stack, you need to save the images again to update them and when using the software when you save, it only saves the top image of the stack and not the whole stack, or it tries to save the whole stack as a file instead of updating your images. So, if you aren't getting such a message that your images are of different sizes in the stack, then you could just use the images at that original dimension. Unless of course, you have some reason for wanting the images specifically in 1024x768.

If these aren't the problems, I could try taking a look at the images themselves and see how it looks on my end of things when I train a classifier and try to run the software. For example, 3 images before and after you processed them. This is of course if you don't mind sending the images, I understand if you wouldn't want to.

I hope this helps, I'll be more responsive with these errors now that holiday break is over :)

Sincerely, Tyler Jang

sbg30 commented 8 months ago

Happy new year!

Thank you very much for your answer, no worries at all for the delay! The reason I was downsizing the images to 1024x768 was because otherwise I would encounter a java heap space error. But you are right, so to avoid any reformatting of the size, I used the original images (2048x1536), I just used less to create the classifiers.

Unfortunately, I ran into the same error as previously described.

I would love to have your opinion, these are generic images to set up my workflow so I don't mind at all sharing them

I have sent you the original files (.jpg) and the transformed to 8bit tif files (3 pictures in each format).

Thank you again for all your support,

Best

Simon

TylerJang commented 8 months ago

So I tried to generate classifiers and run ACCT 1 on the 3 8-bit tif files that you gave me and the software works on my end without the error message. I made 2 classifiers and for the sake of testing, I put the 3 tif images into training_area/validation_images and ran ACCT 1, which gave me output images in training_area/weka_output which means that the classifiers were applied. I can send the exact classifiers I generated to your email since Github doesn't allow me to upload the file type.

Hmm, now that you mention heap space errors I do recall an error when making this software where I'd put too many images at one time into training_area/validation_images which would cause issues. The solution was to run the program in smaller batches of images which should methodologically be equivalent to running the whole batch of images at once so long as the classifiers and parameters are the same.

I wonder if there is some other installation error as well? Have you tried using the github's demo branch of this project and see if works fine? One thing we specify in the download, is to use exactly Version v1.53c of the Fiji distribution of ImageJ and we provide a link to that specific version in the README. This is because there was an issue with later versions of the Fiji distribution that caused load_classifier to fail which we have since asked the owners of the repository to see if they can fix but we are still waiting on a response.

Here's the mentioned demo branch. You can find it by clicking on main and going to the other branch. image Then you can download that branch from the zip folder or by command line.

TylerJang commented 1 week ago

Marking as resolved due to time :)