search

RockStarCoders / alienMarkovNetworks

Using MRFs and CRFs for computer vision problems.
21 stars 9 forks source link

find out the standard training-test split for MSRC #22

Closed jsherrah closed 11 years ago

jsherrah commented 11 years ago

or are people testing on the training data? What are the common hacks to make results comparable. How is accuracy measured

jsherrah commented 11 years ago

Here are the things we need to know for MSRC experiments in the literature:

  1. what selection of images is used for training, validation (hyper-param selection) and test.
  2. is it divided up by image? or fraction of all superpixels?
  3. any smarts to keep class distributions the same?
amb-enthusiast commented 11 years ago

I think Shotton's "TextonBoost for Image Understanding" ICJV 2007 paper introduced the MSRC dataset to the community. Shotton published the split used in there experiments: http://jamie.shotton.org/work/data/TextonBoostSplits.zip

1) Klucker et al [Super-Pixel Class Segmentation in Large-Scale Aerial Imagery, 2010] et al state: The dataset includes 532 images with 21 object classes. As suggested in [Shotton et al TextonBoost, 2007], 276 images are used for training and the remaining 256 for testing.

2) Gould et al [Multi-Class Segmentation with Relative Location Prior, 2008] states: Following the protocol of previous works on this database [22, 24, 31], we ignore void pixels during both training and evaluation. We over-segment our images using the method of Ren and Malik [20], tuned to give approximately 400 superpixels per image. We randomly divide the images into a training set with 276 images and a testing set with 315 images, allowing us to be directly comparable to the results of Shotton et al. [24 - Shotton TextonBoost, 2007].

I'd say it is safe to follow in their footsteps and use the "protocol" defined in Shotton's 2007 TextonBoost paper for dealing with the MSRC Dataset.

amb-enthusiast commented 11 years ago

TextonBoost paper "protocol":

1) 591 images

2) 21 object classes: building, grass, tree, cow, sheep, sky, airplane, water, face, car, bicycle, flower, sign, bird, book, chair, road, cat, dog, body, and boat.

3) Split ratio: Approx. 45% training, 10% validation and 45% test sets, approx. proportional contributions from each class.

See http://jamie.shotton.org/work/data/TextonBoostSplits.zip

jsherrah commented 11 years ago

What are the excluded classes? Which images do they appear in?

Is the class exclusion automatic, in that if we use the given images, those classes won't appear? If not, then we have two choices:

  1. hack in some code to treat those classes as void (yuk)
  2. remove the classes from the list, and treat any unrecognised classes as void. (bit nicer, but silent failure - yuk)