Order of labels differs across categories

[ferewi]

Hi,

while using your pretrained models (thanks again for providing them) and trying to get to the same F1 scores on predictions you published I came across the following issue:

The order of labels in the csv files provided in the "labels" directory differs between some categories. Taking the order in "airplane.csv" as the correct order (alphabetical), I found the following differences using this bash command: $ for lfile in ./*; do echo $lfile; diff <(head -n1 airplane.csv) <(head -n1 $lfile); done

• desert.csv
  • "chaparral" moved between "water" and "wetland"
• golf_course.csv
  • "golf course" moved between "water" and "wetland"
  • "grass" moved between "transmission tower" and "trees"
• intersection.csv
  • "buildings" moved between "factory" and "field"
  • "cars" moved between "parkway" and "pavement"
  • "crosswalk" moved between "road" and "roundabout"
  • "trees" moved between "grass" and "greenhouse"
  • "intersection" moved between "trees" and "water"
• island.csv
  • "trees" moved between "golf_course" and "grass"
  • "island" and "sea" moved between "trees" and "water"
• parkway.csv
  • "parkway" moved between "trees" and "water"
• railway_station.csv
  • "pavement" moved between "bulidings" and "chaparral"
  • "cars" and "parkway" moved bewteen "park" and "parking lot"
• storage_tank.csv
  • label name "snowberg" is missing (blank cell)
• wetland.csv
  • "grass" moved between "track" and "transmission tower"
  • "trail" moved between "golf_course" and "greenhouse"

This screenshot illustrates the issue

I got aware of this issue as I had some labels which have precision, recall, f1 and accurracy scores always being 0.0. Those labels are: crosswalk, intersection, island, stadium, tanks.

What I already tried is to reorder the columns in the csv files in alphabetical order. But this did not improve the results - it just changed which categories receive 0.0 scores (e.g. crosswalk is now fine but desert is at 0.0)

Could you tell me, what you did to train your models? Did you use different label files? Or did you evaluate categories separately?

Any information leading to the reproduction of your results is very welcome. I appreciate your help.

Cheers Ferdinand

[cugbrs]

Hi, Ferdinand

Thanks for your attention.

We have checked our uploaded label files and found the issue you mentioned. Now, we have uploaded the correct version of these files. For models, we are testing and will respond to you later.

With kind regards,

Xiaoman Qi

[ferewi]

Hi Xiaoman,

thank you for providing the updated label files. Using them, the predictions look better, but I still get an f1 of 0.00 for "desert" and "golf course" and quite low values for other labels related to the "misordered" categories (like crosswalk, island intersection). Thanks for looking into the models again.

I am looking forward hearing from you.

Cheers Ferdinand

[cugbrs]

Hi Ferdinand,

we have completed testing of the model. The results of the classification based on the DenseNet169 model are shown below. It is worth noting that we have used 10% of the data for the test in this experiment.

After replacing the model with VGG16, we did find cases where the precision and F1 of individual labels were 0. We believe that this is due to differences in the ability of the models to extract features. In addition, it is possible that the classification accuracy is lower due to the relatively large intra-class variation and complexity of features in some labels.

With kind regards, Xiaoman Qi