Problem with transfer of classification from stack to images

[christianrickert]

Hello, there's something curious about the transfer of my classification model from a training stack to my experimental dataset: The probability map for the stack looks fine, but the probability map for a single image from the same stack looks different - using the same model for classification.

@iarganda suggested in a response to another issue to train on either a stack or on a sequence of images (for multiple images). I've trained on a stack with 30 images depicting nuclei - using fairly computationally expensive features.

Here's the probability map for the first image of the stack:

And here's the probability map for the same image extracted from the stack:

The segmentation results for both images differ significantly, even though the same classification model has been used.

The macro below will create the probability maps for the training stack and then duplicate the first stack image and create the probability map with the same model for comparison: it takes roughly 25 min on my system with a peak memory usage of about 13.5 GB.

// load image stack
file = File.openDialog("Select example.tif");
title = File.getName(file);
folder = File.getParent(file);
classifier = folder + File.separator + "example.model";

wekaversion = "v3.2.34";
run("Bio-Formats Windowless Importer", "open=[file]");

// create probability map for stack of images
selectWindow(title);
run("Trainable Weka Segmentation");
selectWindow("Trainable Weka Segmentation " + wekaversion);
wait(5000);
call("trainableSegmentation.Weka_Segmentation.loadClassifier", classifier);
call("trainableSegmentation.Weka_Segmentation.getProbability");
selectWindow("Probability maps");
rename("Probability maps - training stack");
close("Trainable*");

// create probability map for image (from stack)
selectWindow(title);
run("Duplicate...", "title=image");
run("Trainable Weka Segmentation");
wait(5000);
selectWindow("Trainable Weka Segmentation " + wekaversion);
call("trainableSegmentation.Weka_Segmentation.loadClassifier", classifier);
call("trainableSegmentation.Weka_Segmentation.getProbability");
selectWindow("Probability maps");
rename("Probability maps - image from stack");
close("Trainable*");

The files required for reproduction of the issue can be downloaded here: example.zip (MD5: EA367E1226010A1E14B3A1570DA2DD5F)

As a consequence, I can't train on a larger dataset to improve the segmentation results.

Any idea what went wrong with my approach or what I could do to prevent this issue in the future?

[christianrickert]

I've done some troubleshooting. My current working hypothesis is that the Gabor training feature is not compatible with stack training for single image classification. As before, I've trained on the large stack with the same training features - excluding Gabor!

Here's the probability map for the first image of the stack:

And here's the probability map for the same image extracted from the stack:

The probability maps are identical. Here's the difference (all pixels have a value of 0.0):

IMHO a word of warning might be helpful for users using reading the Trainable Weka Segmentation Plugin documentation: In particular, the section describing the Trainable Weka Segmentation Training features (2D) could include a sentence highlighting the stack issue.

Is this issue known or even by design? I'm not experienced enough with the feature implementation...

[iarganda]

Thanks, @christianrickert for reporting! This looks definitely like a bug in the calculation of the Gabor filters. I'll have a look and be back to you!

[christianrickert]

Thanks for looking into this issue @iarganda. Happy Easter Monday!

[iarganda]

Dear @christianrickert,

Sorry for the late replay. I haven't been able to reproduce the error using my own images (and the link to yours has expired)...

ignacio

[christianrickert]

@iarganda No worries, - I've uploaded the TIF again:

nu-stack.tif (MD5: 42B925923D8C7CE34453416A7C19F78A)

[iarganda]

@christianrickert I'm still unable to reproduce the error. Here is what I've done:

1) Train TWS using a substack of your nu-stack.tif with only 2 slices (for simplicity) using Gabor features only. 2) Save classifier and output probabilty of the whole stack. 3) Run a fresh TWS with the substack, load the classifier and get the probabilities (same result as before). 4) Run a fresh TWS with only the first slice of nu-stack.tif (or substack), load the classifier and get the probabilites (same result as the first slice of 2 and 3.

Is there anything else you are doing?

[christianrickert]

@iarganda Thanks for working on this issue!

I think one of the issue with reproducibility might be that you used the Gabor feature only (I didn't). Another one is that I didn't run a fresh TWS with the sub-stack, nor did I run a fresh TWS on the first slide.

Let's make sure you can reproduce the issue first. I've uploaded the macro, the stack, the model, and the traces I've used again: example.zip (MD5: 995C8BBCAF34B6CB2B774331A23D166B)

Please run the script, point the dialog to the stack image and let the script produce the two probability maps. I've just run it with the latest versions of Fiji, Bio-Formats, and TWS and was able to reproduce the issue.

[iarganda]

OK, I found the problem and it was not an easy one! When you have single slice, the features are calculated calling a multi-threaded method, while a single-threaded method is called for each slice if you have more than one. Usually there is no problem with that, but Gabor filters take some time and, in your case, the order of features in your final feature stack changes from one case to another... and WEKA classifiers don't like that :(

I will add a method to reorder the feature stack when needed as soon as I can.

[christianrickert]

Awesome work @iarganda!

Out of curiosity: Would it be possible to continue using the multi-threaded function call (with all threads) for the stack and iterate through the stack one slice at a time with a fixed order? - In your proposed approach (as far as I understand the issue) the number of active threads would be limited by the number of available slices in a given stack.

[christianrickert]

The update fixes the issue #63. Thank you very much @iarganda!

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/trainable-weka-segmentation-could-not-apply-classifier-error-after-classifier-load-with-anisotropic-diff/28400/2