mayacakmak / se2

Control interfaces for manipulating SE2 configurations
BSD 2-Clause "Simplified" License
1 stars 0 forks source link

Systematic data collection #7

Closed mayacakmak closed 4 years ago

mayacakmak commented 4 years ago

The practice screen randomly specifies the target after each trial. For the systematic data collection we need to systematically vary:

We need to keep the total duration of the experiment within a certain range to avoid fatigue, so we can decide how many variations of each of these we want to do based on how long a single session tends to take for the slowest interface.

Sampling variations: Rather than having fixed/exact variations we might want to have a complete set of exact options and add a little bit of noise to everything (center_x, center_y, center_theta, thresh_xy, thresh_theta => note that the variations are in terms of dist_xy, dist_theta, thresh_xy, thresh_theta, but noise in the former would result in less recognizable patterns).

mayacakmak commented 4 years ago

Based on discussion with Kavi during meeting on Jul 23, Thu.


We'd like to systematically vary four variables. For each variable we propose the following values.

Instead we could imagine a 5 by 5 grid at the center of the screen, leaving larger buffers on the sides compared to top/bottom.

Illustrations here: https://docs.google.com/presentation/d/1XOFJU3b2MvQ5-VBSauC5McdfJ62DB-ecwA4WOWg2ZXY/edit?usp=sharing

Between 0 and 180, should be 45 degrees -> 5 target angles (if we include 0). For each angle randomly select sign (+/-).


Random noise: The noise added to dist_xy and dist_theta should be smaller than max_xy and max_theta (45). The noise added to dist_xy should be applied in a random direction sampled between 0-360 degrees to select the actual x,y position of the target. The noise added to thresh_xy and thresh_theta should be within 1/4 of max_xy and max_theta

There should be meaningful difference between the different threshold values, otherwise might be worth increasing screen size.

We should ensure dist_xy=0 and dist_theta=0 does not happen at the same time (object already at target). Further the thresh_xy and thresh_theta should not involve the initial object pose.

We should ensure the target is visible (not hiding behind manipulation controls).


With the above proposal we have 5x dist_xy, 5x dist_theta, 3x thresh_xy, 3xthresh_theta, which results in 225 different unique configurations. This would require splitting the full set of configurations across participants, and further large sample size to get good coverage through the random noise.

A reduced proposal such that all configurations can be covered by a single participant could be 3x dist_xy, 3x dist_theta, 2x thresh_xy, 2xthresh_theta, resulting in 36 configurations. The intermediate values would be covered through the random noise variations across participants. This proposal might result in a more balanced individual experience.

mayacakmak commented 4 years ago

Moving email discussion here for future context

FROM KAVI:

I want to let you know that I updated the flexible position visualization algorithm to reflect what we talked about in yesterday's meeting.

While starting to update the sampling algorithm, I also ran into something that I think might affect the video.

Currently, our workspace is 750px wide and 500px tall. That means that we are losing a lot of space because the size of the circle is limited by height. That combined with the space we lose because of the handles on controllers, means that our final circles are pretty small.

If we were to increase the height 750px (equal to the width) it would give us a much larger area to sample distances from, and a larger range of distances.

image

This image compares two interfaces with the same width, but different heights. The top image is what we currently have. The bottom image has an equal width and height, and the sampling area is significantly larger. image.png If we want to change the size I don't think that it would be very difficult at all, but I assume that it needs to happen before we can finish the video.

Right now, the size of the workspace is still 750x500, let me know if I should change it

mayacakmak commented 4 years ago

FROM MAYA:

Indeed, we are limited by the height of the workspace for sampling target positions. Making the workspace square is a good idea but two concerns: (1) we want to make sure the whole workspace is on the user's screen when they're doing the task (so their time to complete tasks does not include things like needing to scroll); 750 is still a good number even for those small laptop screens, but might require some scrolling to center around the workspace (hopefully before they start the task). Actually, I just tried on my MacBook screen and with the top/bottom buffers of the browser it barely fits. (2) We were hoping to use one of the "empty" sides, where targets don't land, for one of the controls ("panel") (though the current pose of that is the top left corner so we could be fine).

Also, this makes me realize a different point--for the three different dist_xy ranges, we do want the first one (#1 in your plots) to be around "0" center (i.e. no/almost no position change of the object needed)--so you can merge the white area into #1 and re-adjust sizes.

Okay, I guess the question remains: with either option, are we covering sufficient variation in dist_xy? I guess we should think about comparing it to the range of movement we would need in the real interface (e.g. with the three views, gripper being moved). Those windows might need to be even smaller to fit three views of the robot. Typically they are rectangular and arranged in a 2x2 grid in a certain way (similar to CAD software conventions). The idea of having three squares side-by-side is interesting, less conventional but might make sense. I'm quite torn.

Here's another alternative that would still allow uniform sampling of dist_xy but would make some of the hidden variables less evenly distributed (e.g. for higher distances there would be more x movement than y movement). If we are keeping the 'real interface' rectangular I think that that is fine. Also x and y have the same interface in all our control options to we'd expect things to not be very different. What do you all think?

Screen Shot 2020-08-07 at 4 53 09 PM
mayacakmak commented 4 years ago

FROM KAVI:

I see what you mean about a height of 750 being pretty large for a laptop (I usually program on a large display but I just tested it on my laptop). I like the idea in the picture at the end. This is a slight modification that gives us a bit more sampling space for the outermost circle:

image (1)

One interesting thing about how we are sizing the bands is that the distribution will not be uniform (Because the width of each band is the same, the outer bands have a larger area). I assume that we are okay with that, but if not we might want to change the width of the bands to compensate.

I think that I'm leaning towards the solution in the above picture as a nice balance between large enough variation in dist_xy and keeping the workspace a reasonable size.

mayacakmak commented 4 years ago

FROM MAYA:

Great! I like your variation with more options, but not sure how the sampling algorithm would capture those additional regions--so feel free to go with the simpler option (just sampling a dist_xy and an angle from the screen center from the right ranges) if this turns out to be tricky.

And good observation about not uniformly covering the area--that is totally fine, we just want to uniformly cover the dimensions that we think will impact performance (dist_xy, dist_theta, thresh_xy, thresh_theta).

mayacakmak commented 4 years ago

@KaviMD Heads up I'm moving the sampling javascript code into its own JS file and integrating with the actual control display, assuming you haven't done this yet.

mayacakmak commented 4 years ago

Done with integrating config sampling with the test interface. Going through 36 configs a couple times, the sampling feels somewhat uniform :) currently order of the configs are systematic, added a todo in code to shuffle. Everything needs to be tested for robustness.

kavidey commented 4 years ago

I just added some code to shuffle the configs.

While doing some testing I also noticed that there are situations where the displayed flexible rotation does not match the calculated one in code: This target has a threshTheta of 59, but it does not appear to be centered around the target. As long as the EE is rotated above the target, it looks fine: image For some reason when the EE goes below the target, it doesn't count as a correct position: image

This has only happened once or twice, but I figured that it was still good to look out for.

I also noticed that the Arrow Control has an error at the start of every cycle after the first one. Everything appears to work fine even with the error, but I'm going to test with some other interfaces and see if I can tell what's going on.

One other thing is that it feels pretty subtle to me when the button in the bottom right switches from displaying x/36 to Done. Most people would probably still figure out what to do, but given that their focus is already on the workspace, it might be worth it to instead, add the x/36 to the top right corner of the workspace, and add a Done button that takes them to the questionnaire (similar to the I'm ready button at the beginning of each cycle).

mayacakmak commented 4 years ago

@KaviMD I also noticed the same thing once or twice, but could not replicate (or did not try to investigate yet). As in your example, mine was also around 180 degrees (instead it was green above 180 but black below and it wouldn't trigger 'success'). I suspect this is a bug related to trigonometric computations that can get tricky when crossing 180. See if you notice anything.

mayacakmak commented 4 years ago

@KaviMD any luck re-creating/debugging this false range issue?

mayacakmak commented 4 years ago

@KaviMD Okay I just forced the target theta to be 180 and the issue is repeatable every time. I made a screenshot video I can share. That might give an idea for where to hunt for the bug ;)

Screen Shot 2020-08-18 at 1 44 45 PM
kavidey commented 4 years ago

@mayacakmak Thanks! That helps a lot. From what I can tell, the issue is with how the target and EE represent angles. Theta for the EE ranges from -180 to 180 while for the target, it ranges from 0 to 360.

Because of that, if the target has an angle close to 180, and the EE an angle of 170 or -170, in both cases the EE is within 10 degrees of the target, but mathematically only 170 counts as correct.

I updated the isSame function that checks if two poses are overlapping to normalize all angles between 0 and 360 and that seems to have fixed the problem (I tested several angles and thresholds around 180 and they all seemed to work).

mayacakmak commented 4 years ago

Ah, nice work figuring that out!