Directions forward

[ardila]

@yamins81 There are 2 main goals

1) A screening set that is representative of the difficulty in the 1000-way categorization task, for creating a challenge submission

2) A screening set that is representative of the difficulty that humans are good at in all of imagenet, for getting better neural fits

re 1) We should use random L3 models (5 sets of features, one from each random model) and find a set of images that is hard to separate on average for the model class. This would mean extracting #N1 images from each synset, then getting margins for all 2-ways for each image. Then, we could just take the mean of the set of negative margins for each image as a score, and take the #N2 lowest scoring images.

re 2) We should find the largest negative margins as above, but then for each of these margins, test it in humans. This means that we will have ranked list of tuples ranked by margin (most negative first): (image, distractor_synset, margin)

And we will search through this set of image tuples using psychophysics to find the first (going down the ordered list) #N2 tuples that have a performance above some threshold.

Here are some training curve results for MCC2 classification The results for linearsvc are still being calculated (takes about 210 minutes to generate one of these curves.)

Immediate points of action: 1) Deciding how many images per synset to extract (#N1), then extracting them. 2) Deciding the size of the screening set (#N2)

N1 seems to be around 400 given the training curve (saturation around 300-350, need 50-100 test examples)

If you agree with this decision for #N1, then I will create a new dataset called PixelHardSynsets which you should then extract

import imagenet
dataset = imagenet.dldataset.PixelHardSynsets

[ardila]

MCC confusion matrices HMO confusion matrix

Pixel confusion matrix

[yamins81]

Comments:

1) For both of the two options above, we could also replace the "average-L3-hard" score with the "HMO-0" score, correct? By HMO-0, I mean the current HMO model as extracted so far. I haven't yet completely thought through what I think is best here. Or we could do V1-hard or HMAX-hard, right? I am kind of leaning toward HMO0-hard at the moment. What are your thoughts?

2) The method you described might generally be called "worst margin", e.g. you pick images as the ones with the worst margin on a classifier. I think this should be amended in two ways: a) First, we should make sure that any margins are those averaged over a set of splits, so that the "bad images" are truly those that have stably bad margins, regardless of the specific distractors. b) We should include a set of additional distractors that are randomly chosen with respect to margins. The reason to do this is that oftentimes, I have had the impression that "hard images" (or hard objects) are hard because of the "easier" distractors that are also in a given set. In other words, those "easier distractors" being present is important to expose the difficulty of a given "hard" image or object. If we remove all the easy ones, than it might suddenly look easy to solve the hard images, because they get "moved into place" on top of where the easy ones used to be. Then, once we try to combine the solution back in complementarily, it won't work. So, we'll want to keep at least some easy distractors around that are uniformly distributed in image space with regard to margins on the test algorithms (and classes).

3) I assume that you think we should draw the images from which to choose these set from the pixel-hard synsets as opposed to random 250K images. This is why you're saying that we'll start extracting from tomorrow the "PixelHardSynsets" set, right? How many hard synsets are you thinking? Or will that be set by N1 to fix the size of the total set?

--> On a separate note, what we're doing here is basically stacking a

hierarchical series of increasingly stringent tests, to winnow down the set. Starting with pixels, and using that to cut down the set a lot. Then we can cut it down further with HMO0- or HMAX- or whatever we decide on point 1) above. We then run THAT through either HMO procedure directly, or THEN through humans to re-weight it.

4) From the plot you made for the HMO0 model, I don't agree that we're seeing saturation, though, in the performance as a function of training examples. In fact, it looks to me like its slow (approximately logarithmic) increase, much like in the case of HvM. I expect that performance will keep increasing slowly with the number of examples. But I think N1 = 400 is fine, probably, since we don't need to push out to saturation, we just need a representative sample.

5) Does this plan relate clearly to the psychophysics plan you came up with a couple of months ago? Can you spell that out a little more explicitly now, again?

On Mon, Sep 30, 2013 at 5:47 PM, Diego Ardila notifications@github.comwrote:

@yamins81 https://github.com/yamins81 There are 2 main goals

1) A screening set that is representative of the difficulty in the 1000-way categorization task, for creating a challenge submission

2) A screening set that is representative of the difficulty that humans are good at in all of imagenet, for getting better neural fits

re 1) We should use random L3 models (5 sets of features, one from each random model) and find a set of images that is hard to separate on average for the model class. This would mean extracting #N1 images from each synset, then getting margins for all 2-ways for each image. Then, we could just take the mean of the set of negative margins for each image as a score, and take the

N2 lowest scoring images.

re 2) We should find the largest negative margins as above, but then for each of these margins, test it in humans. This means that we will have ranked list of tuples ranked by margin (most negative first): (image, distractor_synset, margin)

And we will search through this set of image tuples using psychophysics to find the first (going down the ordered list) #N2 tuples that have a performance above some threshold.

Here are some training curve results for MCC2 classification The results for linearsvc are still being calculated (takes about 210 minutes to generate one of these curves.)

[image: screen shot 2013-09-30 at 5 22 44 pm]https://f.cloud.github.com/assets/2701347/1241139/cdc9188e-2a17-11e3-99a5-c8acd6783cb1.png

Immediate points of action: 1) Deciding how many images per synset to extract (#N1), then extracting them. 2) Deciding the size of the screening set (#N2)

N1 seems to be around 400 given the training curve (saturation around

300-350, need 50-100 test examples)

If you agree with this decision for #N1, then I will create a new dataset called PixelHardSynsets which you should then extract

import imagenetdataset = imagenet.dldataset.PixelHardSynsets

— Reply to this email directly or view it on GitHubhttps://github.com/ardila/behavioralthor/issues/2 .

[ardila]

Some vocabulary: challenge subset -> dataset for goal one imagenet subset -> dataset for goal two

1)

The various options are Just pixels V1 <- probably will require some engineering effort/setup time from me Hmax <- probably will require some engineering effort/setup time from me V1+HMax <- probably will require some engineering effort/setup time from me Random L3 HMO The problem with HMO hard is that if we believe that HMO is capturing key axes of difficulty, then we will be removing those from the dataset. This is ok if we have some principled way of combining the model we screen on the challenge subset with our existing model, but even if we do, at some point we should think about regularization (how many times is it fair to screen on a new dataset and add more components to the model). If we are not combining models, then we want to remove only the axes of difficulty that will automatically be captured by almost any member of the model class, which is why I suggested random L3s

2)

a) agreed b) Once we have a set of tuples with high deltas: (image, distractor_synset, delta = margin-human performance as margin (using logistic regression)) We can construct the imagenet subset in several ways, here is one suggestion: If we think of the deltas as weights, then every distractor synset will have some amount of weight summed over all the tuples. We should take a random sample of images from each synset whose size proportional to the synset's weight. There is now one free parameter: # hard images/# images from distractor synsets which can be set empirically to ensure that the screening set is actually difficult for HMO-0.

3)

It depends on N1. Since we've agreed N1=400 is ok, the number of synsets depends on the budget for extraction which you said was 250,000 (833 synsets). If that is correct, then you should begin extraction of PixelHardSynsets ASAP (should be ready in 15 minutes from the time I post this)

[ardila]

PixelHardSynsets is now available: e93d9e99547c2fe05e48d264bf9219589ca9bc54

Here are svm results (not much different from MCC results):

I am also running the following classifiers using compute metric base: 5-NN and SGDClassifier

[ardila]

[ardila]

@yamins81 In talking with Jim about priorities, I think we came to the conclusion that we need to take advantage of the work I've done so far in some way, instead of just dropping it all to move to a new problem. Looking through what I have I was wondering if you still thought that "finding the hard parts of imagenet" is a useful goal.

I'm pretty convinced that I've done this: I have all 2-ways of the best model I can run, and found the densest part of the space. I have measured the human and model performance at just a few points in this space and it looks like there is a significant gap with humans (just not in 2-ways because humans and models are near ceiling). If you are not convinced of this gap, what would it take?

Is it possible to run the HMO procedure again on a combination of however much of this dense space would be appropriate + the synthetic set from before?

At the very least, I want run some sort of apples-to-apples comparison on Imagenet with HMO and the others, especially since I've found that

1. The gap on HvM is still significant, and here HMO is the most consistent with humans
2. The consistency between humans and the convnet models is generally low on imagenet, especially in the dense subspace.