The new sampling strategy has greatly reduced the number of training and test crowns. There are only ~ 200 test crowns and probably 1/3 of them already have polygons from field surveys. I am somewhat hesitant to spend time hand labeling them.
Pros:
We clearly see examples of poor crown delineation in the confusion matrix among species that should not be confused. This
NYSY (our lowest accuracy species) crown is terrible. This is as bad as i've seen.
In cases like this we can clearly draw a better crown. It is reasonable to hypothesize that model performance should improve.
Cons:
One of my guiding principles is that we never want to do anything to our test data that we cannot do during prediction. It is natural to want to make the evaluation score higher, but if cannot be connected to prediction, then it represents more of a self-serving desire for higher numbers than anything that will help others. The counter argument is that without identifying the problem, we cannot improve it. If the model does significantly improve through crown delineation that makes a compelling case for next steps, if it is only minor improvement, then it is also reassuring. It also is fairly trivial to code evaluation both ways, or take the centroid of the polygon and present both numbers.
It may bias us towards highly visible crowns. We are tasked at predicting all trees in the landscape, not just the big ones you can see.
Given where we are in the project lifecycle, I'm probably going to do this, but just fleshing out my thoughts for when @ethanwhite is back.
The new sampling strategy has greatly reduced the number of training and test crowns. There are only ~ 200 test crowns and probably 1/3 of them already have polygons from field surveys. I am somewhat hesitant to spend time hand labeling them.
Pros:
In cases like this we can clearly draw a better crown. It is reasonable to hypothesize that model performance should improve.
Cons:
One of my guiding principles is that we never want to do anything to our test data that we cannot do during prediction. It is natural to want to make the evaluation score higher, but if cannot be connected to prediction, then it represents more of a self-serving desire for higher numbers than anything that will help others. The counter argument is that without identifying the problem, we cannot improve it. If the model does significantly improve through crown delineation that makes a compelling case for next steps, if it is only minor improvement, then it is also reassuring. It also is fairly trivial to code evaluation both ways, or take the centroid of the polygon and present both numbers.
It may bias us towards highly visible crowns. We are tasked at predicting all trees in the landscape, not just the big ones you can see.
Given where we are in the project lifecycle, I'm probably going to do this, but just fleshing out my thoughts for when @ethanwhite is back.