enhanced tracker pose calculation

[gb2111]

it is known issue that in case of 3 or 4 cameras when one camera is losing bulb then we have tiny jump. this is caused by fact that position is calculated as average of individual interpolation of each available tracker pair. so if we have 3 trackers and we hide 1 then instead of (p12+p23+p13)/3 we have p23.

me and @zelmon64 had discussion about algorithm where we could correct error by p23 + err where err = (p12'+p23'+p13')/3 - p23' and p12', p23' and p13' is some kind of interpolation of all tracker positions collected either during gameplay or calibration.

i did simple test of algorithm just remembering last point where I has all trackers and results were not bad. however when I moved away from point and then gained tracker it was not.

the simple algorithm could work like that but be active in only short distance while full algorithm could be more complex and collect points across whole game area and interpolate from points that are close enough to actual position also over whole area. so if user lost one tracker in one please, the position of the controller would be corrected all the time until all trackers.

second approach would be to extrapolate instead of interpolate from 5 points taken during mat calibration. for each point we can calculate err and potentially find a function that will calculate err in new point from calibration point err and distance from center of area.

here, we assume that it is unlikely to calibrate cameras so well that (p12+p23+p13)/3 = p23. I would say its not possible but not sure.

@cboulay , @HipsterSloth , would you be so kind and give us your comments? what approach you would recommend? or there is alternative way.

Greg

[cboulay]

I'll need to find some time to look this over.

However, I want to say that I think the next thing to modify in the optical tracker is to make each tracker have its own thread and behave asynchronously. This deserves its own issue and discussion so I won't outline it here. Once that work begins, pretty much any changes to the tracker will conflict so all such changes should be put on hold.

[gb2111]

agree. this will introduce much conflict in code. can you tell if orientation data will also go to separate thread? that would be really great if we could use maximum of it. is this something that will be implemented in near future?

[cboulay]

I wasn't planning on starting work on it until April. So I guess you have a couple weeks to make changes to the tracker code. But this particular issue would probably need a different approach once the threaded trackers are implemented so it's probably best to hold off.

By the way, take a look at Matlab's triangulatemultiview for triangulation with more than 2 cameras.

[gb2111]

Note only that it is not to enhance triangulation but handle case when one camera does not track object. I don't see such concern in matlab fun.

[cboulay]

I don't really understand the sub-scripting in your original post. Are the numbers after the p the camera IDs? If you drop from 3 cameras to 2, how do you still have p12 and p23?

The Matlab function gets a single 3D position from multiple cameras, without first calculating 3D positions from all pairs of cameras then averaging those. So, your (p12+p23+p13)/3 wouldn't exist, there would simply be p123. Then when one camera drops out it would be p12 (or p23).

As to whether or not you can construct an error model that describes the difference between all p123 and all p12 (or p23)... I don't know. I imagine it is very non-linear and difficult to describe analytically. Can you collect a bunch of data points from your volume and share it with me? Each row in the table should have 5 columns per tracker: 2D-x, 2D-y, 3D-x, 3D-y, 3D-z. I can then try to visualize what the error is like if one camera drops out and if it is systematic. I only have 2 cameras so I can't do it.

[gb2111]

U are right. I was too fast If u have 3 cameras then either I have 3 interpolation or one. I have idea how to visualise current error so will post video soon.

[zelmon64]

@cboulay I agree that it would probably be very non-linear. That is why I think we would need a lot of calibration points so that we could interpolate from the surrounding ones. I think it would turn the tracker pose calibration into something similar to how motion capture software move a calibration wand throughout the entire tracking space to calibrate.

[cboulay]

Has anyone tested whether triangulation (dual or multicam) is in fact better than (weighted?) averaging of 3D positions from each camera? The answer to this question doesn't affect the current issue of how to compensate for a lost camera, but it does affect what baseline data and 3D position estimation method I/we use to build the error model.

@zelmon64 , depending on how many points are needed, the error model can become very large very quickly. Let's say you start with 4 cameras, you need 14 different error models (4@3cams; 6@2cams; 4@1cam).

[gb2111]

I will make video and we will see how it behaves. If model will be too complex then we can always collect points and interpolate. But it would be really many points...

[HipsterSloth]

I don't think averaging the conic fit 3d estimations (instead of averaging the triangulations) has been tried. It would be pretty straight forward to add an option for that. My gut says that it would feel worse since you'ld be averaging in more noise (z-noise of conic fit vs z-noise of camera transform error), but I would love to be proved wrong.

EDIT: typos

[HipsterSloth]

Was doing some random google searching for multi-camera triangulation and came accross this book posted online. Pasting here for reference:

http://cvrs.whu.edu.cn/downloads/ebooks/Multiple%20View%20Geometry%20in%20Computer%20Vision%20(Second%20Edition).pdf

[cboulay]

@HipsterSloth With Matlab's triangulatemultiview algorithm, which was taken from the textbook you just linked, you wouldn't be 'averaging' triangulations anymore but it is a least-squares fit which I guess is similar.

But our triangulation (whether multiview or not) does rely on the center of gravity calculation for each camera which is itself is sort of a noisy average.

I think it's worth testing.

[HipsterSloth]

@cboulay It just occured to me that one advantage of averaging the conic fit results is that it wouldn't suffer from the singularity we get that triangulating opposing (or nearly opposing) cameras. It would be nice to eliminate the opposing camera filter. Also it would be nice to only have to do N averages instead of N*(N-1)/2 averages (not that N>4 is really practical for most people given USB bandwidth/processing restrictions).

[gb2111]

I am lost. Will that help with this issue of losing tracker?