Improvements to HvM Latent Variables Tasks

[yamins81]

Suggestions from @hahong : -- in size task, normalize by maximum extent and not radius -- the minimal area bounding box -- segmentation task [ONLY DO LATER] -- pixel mask version of the size task [ONLY DO LATER]

Suggestions from @judithfan: -- reduce lags between stim presentation!!! fix this --longer ISI --surface texture doesn't come immediately --better instructions about angle and real size and depth --move submit button to near the bar.

-- put learning phase in position and pose tasks -- correct trial Counter text given the learning period -- make sure things in the right places in all the images -- polish instruction language -- decide on things like how many images to repeat, how many images per hit, how many hits, which specific variation levels, and other details -- replace objects that have too many polygns -- test small versions all the way through on various browsers

Suggestions from @dicarlo:

(1) have a total of 120 trials per hit (2) in terms of repeats: repeat 20 images twice (3) do all variations together, without blocking of any kind (4) do stimdurations of 100ms and 500ms (5) Position task: -- auto height in instructions -- put marker up during learning images as well -- remove white square in canvas and replace with nothing (just gray background) (6) axis bounding box: -- gray background as with position task (7) area bonding box: -- showing bounding box during learning -- spelling mistake in instructions -- markers too small in clicked indications (8) size: * mark the correct size, as opposed to moving automatically to it during learning * grey background * bonus only to 2 digits * slider bar in correct range (9) pose: -- move submit response button to bottom

[hahong]

Here's my suggestions:

1. General * Announce the beginning of the training and (especially) the end of the training. Minimally, you can add a ~2s slide that says e.g. "Beginning/End of training." Probably it would be better to add a short description about correct way to answer the questions.

   * The first 10 or 20 trials out of 100 are used for training leaving only 90 or 80 usable trials. Make sure you collect good/equal amount of data for all images by e.g. allocating more HITs.[Or, making HITs of length 120...]

   • Did you randomize the image presentation duration? Some presentations looked far shorter than 100ms. This happened during pose estimation task and position estimation task. [NO I did not randomize this, what's going??.]
   • Related, have you somehow randomized inter-trial intervals (= fixation dot presentation duration)? It seems that occasionally there's some variability. In itself it's not a problem. But if you hadn't intentionally randomized we need to worry about this timing issue. [Again, NOI definitely did NOT randomize this. This is something weird we have to track down. I don't experience it in my browser very noticeably.]

   * Some images were repeated at least twice in tandem. If you want to repeat, I would separate them.

   * Bonus should be computed without the answers to the training trials. [There are reasons to do this, however]

   • Some WebGL objects (e.g., boat, table, chair) are still not rendered with correct textures. Would be good to fix that.

   * Hide the "Begin!" button until the user dismisses the instruction.

   • (Low priority) Few objects (e.g. ?steam boat) are composed of too many polygons, giving somewhat bad rendering performance. I'm a bit worried in the cases where the user' computer is crappy.
2. Position task *Fixation dot image is far smaller than actual images. [has been repositioned so that center of fixation image and center of presentation image are aligned]
   • What's the ground true position? It's nominal position or center of mass? (I guess the former.) Don't we want to use the latter? During the training, I was constantly wanting to click on the COM, and the apparent answer looked somewhat wrong.
3. Axis-aligned bbox: * Add "Delete points" button

4. Size task * During training, instead of giving correct answers from the beginning, always start with middle and have the user to slide to the correct position. Mark the correct position with an arrow.

   * Would be better to add few images of explaining this right before starting the actual training (separatetly from the instruction). Clearly indicate the end of training.

5. Pose task * WebGL rendered objects have quite significantly deviated pose compared to the HvM images. Make sure we fix this. See: https://www.dropbox.com/s/ew4lk2873jaj59a/Screen%20Shot%202014-07-10%20at%206.17.57%20PM.png * Similarity to the size task that I wrote above: (1) during the training, always start with random or standard pose and have the user match the correct pose. I think you can provide the presented image sideway. (2) Would be better to give a slide with few images that describe (1) right before the training. (3) Clearly indicate the end of training.

[yamins81]

@hahong Thanks for your comments.

I plan to address them all, and all of them are easy to address except the following:

(1) the timing issues that you noticed. In fact, I did not randomize either the ISI or the stimulus duration. ISI is 500ms and stimulus duration is 100ms (except during the learning period, where it is around 2000ms). If there is variability in either of these, it is due to javascript timing issues that I don't have any control over. This is a significant problem, but I think it would affect all such tasks equally. @hahong do you reliably experience it on just the position and pose tasks?

(2) The issue of what the definition of ground truth in the position estimation task. I am using the COM definition (e.g. "centroid") from the pixel masks, not the nominal position. Why do you think it's using the nominal position? Can you give an example of an image where the "correct answer" as indicated by the system is clearly far from the onscreen centroid? (That is, the centroid of the pixels indicated as 1 in the binary pixel mask for that image. )

(3) Issues with canonical angles in the pose task. There are two issues here. One is that, even when there is a perfect match between the original HvM model and the 3d mode being rendered to match it, I sometimes picked the "canonical angles" incorrectly. The example you give with the car is basically of this form. This I think can be fairly easily corrected by hand. The other problem is harder, which is: sometimes, the rendered object is a different 3d model than the one used in HvM (I matched whenever I could but there is deviation). Sometimes, the geometries of the HvM object and the rendered object are sufficiently different that the canonical pose chosen for the rendered object from one point of view looks "right", but when both objects are rotated to a different angle, looks "wrong". For example: the Bear object. The HvM bear and the rendered bear are in different poses. The pose looks sort of correct at Var0, insofar as it's possible to match the two differently-shaped bears at all. However, at other angles outside of Var0, most humans (myself included) will choose different "best match" angles that are specific to the point of view and thus inconsistent with the "best match" at other views. In other words, the different object geometries makes it ambiguous what the "best alignment" between the two meshes, and the answers will end up being inconsistent between angles. This problem can really only be overcome by having better matches to the actual 3d meshes used in HvM. In some cases, we just do not have this. I can try to search around for more models, which seems like the only real solution ... or drop those models out of the task?

[yamins81]

@hahong One more thing: as for computing the bonuses without the including answers to the training trials: I thought about that, but decided to leave it that way it is, e.g. including the training in the bonus. This is because, insofar as it has any effect one way or the other (probably not much, anyhow), it might encourage the people to actually learn the interface during the training trials more accurately.

[yamins81]

Replace these models lioness with lo_poly_animal_LION_M blcow with lo_poly_animal_SHEEP

MB27840 suport_ship _item_37 _item_11

[hahong]

Regarding this:

One more thing: as for computing the bonuses without the including answers to the training trials: I thought about that, but decided to leave it that way it is, e.g. including the training in the bonus. This is because, insofar as it has any effect one way or the other (probably not much, anyhow), it might encourage the people to actually learn the interface during the training trials more accurately.

Then I think you should also display the amount of expected bonuses during the training phase as well (which is currently not).