switching classes from pen label upon segmentation?

[xandra22]

classes.txt

Hi, I'm on a project where we're using doodler to segment plover habitat in Cape Lookout. We ran into an issue where the doodled mud-peat class is seemingly being swapped for sand upon segmentation. I've uploaded the classes.txt file, the image, and a couple screenshots describing the problem. We have two doodlers working on the project, and both of us have independently encountered the issue. Any help with this would be greatly appreciated! Thanks, Xandra Evans

[ebgoldstein]

Hi @xandra22, Q for you - do you remember if these doodles were done all at once? or were they done incrementally (i.e., Doodle a bit, then hit 'segment', then doodle more, then hit segment again)?

My experience has been that sometimes if i develop a model incrementally by adding more doodles, then a class might not be well-defined by the model that is working behind the scenes. As a result there can be some problems like the one you are presenting (its mud-peat, it is labeled as mud-peat, but Doodler keeps labeling it as sand)..

I have found that errors like this can be corrected by restarting Doodler, and then making sure to label that area as mud-peat (and doodling as much of the image as you can) before you hit the segment button for the first time. So is it possible for you to perhaps restart doodler and re-doodle the image and report back?

[dbuscombe-usgs]

I did some trials to help understand this issue.

1. Label mud and background. Works well

2. Label mud, sand, and background. Works well

3. (really quick/ rough!) label of mud, sand, herbaceous, and shrub-forest. All doodles made in one go. Works well

I assume you are using appropriately low values for downscale (both classifier and CRF), and also normal values for other hyperparameters. If so, I'm inclined to agree with @ebgoldstein that an overly incremental labeling approach may be the culprit.

[xandra22]

Thank you both for your suggestions, I'll restart doodler and report back soon! I'll try to doodle as much of the image in one go, as well.

[xandra22]

After restarting doodler and making sure to doodle as much as necessary to get a good segmentation on the first try, the classes were labeled as expected. I think you were right about the incremental labeling approach - good to know! Thanks so much for the help!

[xandra22]

Apparently I closed this issue too soon - Kathy let me know that this issue is still happening for her even after restarting doodler and not using an incremental labeling approach. Apparently the first image was doodled correctly but then the following images swap the classes? I'll have Kathy follow up with a better description of the problem.

[dbuscombe-usgs]

Please can you provide the offending image(s) and doodle (npz) files, thanks!

[kweber1geo]

Xandra described the issue perfectly. I restarted doodler and did not use incremental labeling (I always label the entire image before segmenting). The first image was labeled correctly, the second swapped sand for mud-peat. I used default parameters. Here's the first one (this was CLASSIFIED CORRECTLY, with lots of mud-peat) :

This is the second one (CLASSIFIED INCORRECTLY):

I can't seem to attach the npz files (git is telling me that this type of file is not supported). Should I email it?

[dbuscombe-usgs]

Please zip up the npz file and post it here (zipped files work). Can I also see your classes.txt file? Are you always using the same classes.txt file, and are you all using the same classes.txt?

[kweber1geo]

Here's the zip file with the npz for the image with the mud-peat class incorrectly labeled as sand. The classes.txt file is also included. Xandra and I are using the same classes, in the same order in the txt file. Thanks!! Issue31_switchingClasses.zip !

[dbuscombe-usgs]

Thanks, I will try to troubleshoot

[kweber1geo]

Thanks, Dan! No rush....

[dbuscombe-usgs]

Initially, I thought the issue was that you hadn't doodled enough - your doodles are below, and don't cover the full scene

So, I doodled the image myself, trying to give the model a better chance ... but I ran into the same issue!

I played with various parameters, but no joy. I've not seen this before, but you are correct! I will investigate further....

[kweber1geo]

Interesting! Thanks so much for looking into this some more, Dan!

[dbuscombe-usgs]

The problem appears to be with the CRF portion of the workflow. Very rarely, it seems that the CRF will switch an entire class. I am now trying to modify the code to catch when this occurs, and provide a revised CRF solution in these circumstances

[dbuscombe-usgs]

I pushed a new version of doodler .... I couldnt figure out how to force the CRF to not switch classes, at least not in a way consistent with normal operations ... so the code will now throw out the CRF solution if the classes get switched.

I got a good result with the offending image, so please try the latest version and report back?

[kweber1geo]

Will do. Thank you! Do we need a full uninstall before installing the newer version of Doodler?

[dbuscombe-usgs]

git pull should work

[dbuscombe-usgs]

@xandra22 and @kweber1geo I had a chance to revisit the doodler implementation this week and I'm pleased to report that the latest version of dash_doodler (combined with the latest doodler-engine) seems to have resolved these issues. The cause (see https://github.com/Doodleverse/dash_doodler/issues/42) was the MLP solution randomly failing, and also switching classes. I hope these issues are now resolved