OSC Output implementation

[GoogleCodeExporter]

It would be great to output the coordinates of each dot being tracked via the 
OSC protcol, opening up the software to a new audience of high end developers. 
This would mean that software like OSCulator, MaxMSP, VVVV, Pure Data etc would 
be opened up to this amazing tool!

OSC:  http://opensoundcontrol.org/introduction-osc

An ideal message format for the OSC message being output from Linux Track would 
be something like this:

dots/dot identifier number/x coordinate/y coordinate/z coordinate 

So for dot number 3 this message would be formatted like this:

dots/3/0.8/0.23/0.9

Similar messages output for each of the other identified dots.

Here I have converted the dot coordinates to a float with 0. being the minimum 
and 1. the maximum. This would be the ideal as it would make it very easy to 
then scale this information in other environments.

It would be ideal if dots were able to maintain their identifier number, i.e if 
three dots have been identified and dot 1 becomes masked for some reason, then 
the software would reassign the same identifier number when the dot gets 
"rediscovered" by the software. 

Thanks ever so much!

Original issue reported on code.google.com by barryfar...@gmail.com on 9 Dec 2014 at 8:24

[GoogleCodeExporter]

Hello,
looking around a bit more, I discovered a liblo, which is lightweight OSC 
library, which may simplify things...

As for the message, I have to clarify one thing - I can either output an [X, Y] 
coordinate of each point the tracker see, or using the 3 point tracking 
(TrackClip, ...) I can give you one [X, Y, Z] coordinate of the head 
(translations) and [Yaw, Pitch, Roll] angles... Of course I can bundle all the 
info into one message too...

As for the dot identification, I do that already (to a certain extent) - for a 
3 point track clip I report the dots based on their Y axis value; for a 3 point 
cap I report the upper, then the one on the left and then the right one (as 
viewed by camera).

Last, but not least - are you targetting Linux or MacOS X? Not that it would 
make difference that big, just when I'd send you something to test, I'd need to 
know what to send you...

Kind regards,

Michal

Original comment by f.jo...@email.cz on 9 Dec 2014 at 8:49

• Changed state: Accepted

[GoogleCodeExporter]

Great to hear back from you and so awesome to think that this might happen! So 
thinking again about the message, and realising that Z is derived from the 
proximity of the dots to one another, perhaps several messages might work 
best...

For each point:

"points/point identifier number/x coordinate/y coordinate"

So for point 3 this message would be formatted like this:

"points/3/0.8/0.23"

for the head:

"head/x coordinate/y coordinate/z coordinate/pitch/roll/yaw"

pitch roll and yaw could potentially range from -1. to 1. ?

I'm using MacOS X Yosemite. 

Very best wishes, 

Barry

Original comment by barryfar...@gmail.com on 9 Dec 2014 at 1:32

[GoogleCodeExporter]

Hello Barry,
I was looking at the liblo library and so far it seems that it will simplify 
things quite a bit...

As for the Z axis, the math behind is a bit more complicated (I use an 
iterative method to get the full perspective solution), but there are two 
things you should be aware of - first of all, the translations are available 
only for the three point models (they give you full 6DOF), single point maps 
them to pitch/yaw (only 2DOF); second thing is, that Z axis value is the least 
precise - it is most sensitive to the blob position measurement errors (it 
needs more smoothing than others)...

As for the pitch/roll/yaw, their values are in degrees (translations in 
milimeters), however bear in mind, that their values can be influenced by the 
tracking setup (limits, sensitivity curves, ...). That said, normalizing them 
is no problem...

I'll keep you posted.

Kind regards,

Michal

Original comment by f.jo...@email.cz on 10 Dec 2014 at 8:43

[GoogleCodeExporter]

Hey Michal,

I totally get what you're saying about the three point model providing 6DOF and 
one point model providing 2DOF. Could the OSC message still be formatted in the 
same way for both one and three point models though? i.e "head/x coordinate/y 
coordinate/z coordinate/pitch/roll/yaw" So one point is formatted in the same 
way as three points but obviously only x / y coordinates and pitch / yaw are 
output. Roll and x coordinate would simply return a 0 or nothing at all. 

Getting the location of each point via a separate "points/point identifier 
number/x coordinate/y coordinate" message would be really, really useful. I 
stuck a load of reflective points on front of Linux Track and noticed that it 
was tracking many more than the 3 we are discussing. How many can it track as a 
maximum? It would be good to output location data for as many as possible, just 
with a separate "point identifier number" for each one. 

Sending pitch/roll/yaw in degrees would be absolutely fine, it's super easy to 
rescale at my end. Regarding the Z axis value being the least precise, no 
worries, the information will still be really useful I'm sure!

If there is anything I can do at my end to help please let me know.

Original comment by barryfar...@gmail.com on 10 Dec 2014 at 1:32

[GoogleCodeExporter]

Hello Barry,
sure, I planned that the message for pose would be same no matter what the 
model is, just the unavailable fields would be zero...

As for more than three points, I think I had there a constant to set the amount 
of points to "register" and I think I used value 10; of course it can be set 
higher if needed... The only worry at this point is the point identifier - the 
thing is, that at the moment I can sort the points according to their X/Y 
coordinates, but I don't have any sort of point position tracking...

For the moment I'm going to concentrate on putting together something you can 
play with and if needed I can try to come up with some point position tracking.

Kind regards,

Michal

Original comment by f.jo...@email.cz on 10 Dec 2014 at 1:46

[GoogleCodeExporter]

Regarding multiple point tracking, at the moment I would only be looking to use 
two points anyway, so if you are able to output the individual positions of 
three points with identifiers then that would be wonderful! It's great to know 
that their is the potential to extend this to more dots at some point but I 
appreciate that is a lot more work. :)

Really excited about the potential of this, I will make sure that I promote 
Linux Track as broadly as I am able to. Like I say, I'm sure that there are 
many other people like me who would be very interested in using it with OSC 
implementation. 

Original comment by barryfar...@gmail.com on 10 Dec 2014 at 3:21

[GoogleCodeExporter]

Hello Barry,
I'm sorry it took me that long, but the Christmass along with the children are 
taking a fair bite off my free time...

Anyway, here is the first shot - in the archive there is a binary of the OSC 
server; it is commandline only (can do a GUI version, when we make it work), 
and it needs a single parameter specifying a port number to sit on. Like:

./osc_server 7770

When started, it will initialize the tracker (assuming you set things in 
ltr_gui already) and then starts pumping out the data. You can control (pause, 
wake, recenter, query status and quit) it using simple "menu" - number followed 
by enter selects the item.

One thing you should be aware of - the pose and the points are passed only when 
valid pose is detected (that is a feature of Linuxtrack interface), so if you 
intend to use different scheme than usual 3 point model, I'd advise you to 
specify a single point model. Then you get all the points, but the pose will be 
2DOF only, based on the first point only...

As for the protocol, I came up with this:

path: </linuxtrack/pose>

arg 0 's' "pitch"

arg 1 'f' -70.415230

arg 2 's' "yaw"

arg 3 'f' 52.430729

arg 4 's' "roll"

arg 5 'f' 0.000000

arg 6 's' "X"

arg 7 'f' 0.000000

arg 8 's' "Y"

arg 9 'f' 0.000000

arg 10 's' "Z"

arg 11 'f' 0.000000

path: </linuxtrack/point>

arg 0 's' "index"

arg 1 'i' 0

arg 2 's' "x"

arg 3 'f' -179.381470

arg 4 's' "y"

arg 5 'f' 219.500946

arg 6 's' "weight"

arg 7 'f' 295.000000

path: </linuxtrack/point>

arg 0 's' "index"

arg 1 'i' 1

arg 2 's' "x"

arg 3 'f' -94.050507

arg 4 's' "y"

arg 5 'f' 129.321899

arg 6 's' "weight"

arg 7 'f' 343.000000

path: </linuxtrack/point>

arg 0 's' "index"

arg 1 'i' 2

arg 2 's' "x"

arg 3 'f' -175.434357

arg 4 's' "y"

arg 5 'f' 96.743958

arg 6 's' "weight"

arg 7 'f' 361.000000

I don't know if that is alright - don't hesitate to let me know if there is any 
problem...

Kind regards,

Michal

Original comment by f.jo...@email.cz on 18 Dec 2014 at 1:58

Attachments:

• osc_server.zip

[GoogleCodeExporter]

Hi Michal, this is awesome, I can confirm that it is spitting out data at my 
end, a massive thank you for making this happen!

Couple of notes regarding the formatting of the messages. It's totally possible 
to work with what you have sent over to me but it might be worth removing the 
symbols from the arguments to streamline things. I would propose this (but only 
if you have the time!)

path: </linuxtrack/pose>
arg 0 'f' -70.415230 (pitch)
arg 1 'f' 52.430729 (yaw)
arg 2 'f' 0.000000 (roll)
arg 3 'f' 0.000000 (x)
arg 4 'f' 0.000000 (y)
arg 5 'f' 0.000000 (z)

path: </linuxtrack/point>
arg 0 'i' 1 (index)
arg 1 'f' -94.050507 (x)
arg 2 'f' 129.321899 (y)
arg 3 'f' 343.000000 (weight)

Obviously everything in brackets would be omitted and could be included in a 
text file somewhere to inform anyone using the OSC output what's what. This is 
so exciting! Again a massive thank you for this. 

Original comment by barryfar...@gmail.com on 18 Dec 2014 at 8:06

[GoogleCodeExporter]

Hello,
here is the new version - I removed the string fields as requested, the rest 
stays as it was...

Now for the future plans, what form would you prefer - commandline tool, or 
some GUI? Or both?
Kind regards,

Michal

Original comment by f.jo...@email.cz on 18 Dec 2014 at 8:49

Attachments:

• osc_server_1.zip

[GoogleCodeExporter]

This is great! I wrote my own osc streamer a couple weeks ago (I guess I should 
have asked!)  based on linux-track and the hello_world examples in the source. 
I'd love to try this out as well. Do you have source available somewhere, or 
would you build it for linux x86_64?

One issue I've found cumbersome in my very simple implementation is that of 
re-centring. I'm using a trackir5 tracker, which should be able to give 
absolute x,y,z coordinates, but I've only found a way to query linux-track for 
coordinates relative to the first detected marker location, or to the location 
of the marker when linuxtrack_recenter is called.

Is there a way to query x,y,z coordinates of the head in the absolute 
coordinate system of the camera? Otherwise, how do you handle sending recenter 
commands to your osc server?

Thanks a bunch.

Original comment by matthew....@gmail.com on 19 Dec 2014 at 9:02

[GoogleCodeExporter]

Hello,
I'm attaching the source - when I get it more polished, I'll add it to the 
Linuxtrack. Anyway, I attach the current source - it should compile on Linux, 
but I didn't test it (it did compile few days ago, but I didn't check after 
making changes needed for OS X). Also make sure you have liblo development 
package installed... Let me know if you encounter any issued during the 
compilation.

As for the recentering and other commands, right now it is controlled from the 
terminal where you started the osc_server (see post #7). I guess I can make the 
streamer capable of duplex communication, so you could send commands from your 
main application...

Now to the absolute X, Y, Z coordinates - right now there is no direct way for 
you to get them; may I ask you what kind of application you'd need them for?
The thing is, that I can come up with a way to get that info to you, however 
knowing the application I might be able to suggest you another way of obtaining 
the information you need...

Kind regards,

Michal

Original comment by f.jo...@email.cz on 19 Dec 2014 at 6:53

Attachments:

• osc_src.zip

[GoogleCodeExporter]

Thanks, I'll take a look at the source.

My application is for head tracking in front of a 3D display, where I have to 
know the correct distance between the viewer and the screen to properly render 
the correct 3D image.

Original comment by matthew....@gmail.com on 19 Dec 2014 at 10:40

[GoogleCodeExporter]

Hello,
at its present state, I'm afraid that Linuxtrack can't provide you with 
absolute coordinates...

There are several things to consider; the method I use to compute the pose uses 
full perspective, but to provide exact 3D position, much more information is 
needed (focal length of the camera lens, lens distortion correction...) and 
given the variety of devices used I don't know how an usual user could come up 
with this kind of info...

The other thing is, that 3 points (and single camera) are the bare minimum for 
the 6DOF tracking, making it quite vulnerable to noise - small changes of the 
blobs position influence the resulting pose considerably (especially the Z axis 
value).
That is the reason why commercial devices use much more points (6, ...).

Let me think a bit more about it and I'll try to come up with a way to provide 
you with the raw position.

Kind regards,

Michal

Original comment by f.jo...@email.cz on 20 Dec 2014 at 9:41

[GoogleCodeExporter]

Michal,

I really appreciate your looking into that. Would you like me to open another 
feature request to keep track of it? With a little work, opencv, and perhaps a 
retroreflective checkerboard, I could estimate the intrinsic matrix for the 
trackir I'm using. I haven't looked into your tracking method in detail. Should 
that be enough information to estimate physical distance?

Thanks,
Matt

Original comment by matthew....@gmail.com on 23 Dec 2014 at 5:37

[GoogleCodeExporter]

Hello Matt,

I'm sorry for the delayed response... 

I'm working on making the absolute position available to you; if things go 
well, in the next few days I should have a test build ready so you can give it 
a try...

There is just one prerequisition - you'd have to use either of the three point 
models (cap or clip), single point or facetracking won't do. If you plan on 
using other model type, in that case you'd have to use just the 2D blob 
positions (as reported by the camera) and do the math yourself... 

That said, if there would be more interest in other type of model, I can try to 
incorporate it in the future...

One more thing - may I ask you what system do you use? I mean what for should I 
do the test build (Mac or Linux; 32 or 64bit)?

Kind regards,

Michal

Original comment by f.jo...@email.cz on 7 Jan 2015 at 6:36

[GoogleCodeExporter]

Thanks very much for looking at this! I've been using the 3 point clip, so that 
will be perfect.

I'm running Fedora 21 x86_64.

Best,
Matt

Original comment by matthew....@gmail.com on 7 Jan 2015 at 6:46

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hi Michal, happy new year! I thought I'd email to let you know what a success 
the OSC implementation has been in our music sessions. We have been working 
with a young man who can only open and close his eyes and raise and lower his 
eyebrows, he can't move any other part of his body. The time you put into 
making this happen has enabled him to contribute in his school orchestra, so a 
big thank you from all of us! I'd like to send a short video of his first 
session to you so you can have a look. Because of safeguarding I'm not going to 
put a link of it on here, do you have a email address i could send it to?

In regard to your previous questions: "Now for the future plans, what form 
would you prefer - command line tool, or some GUI? Or both?" it would be great 
if OSC output could be built into the LinuxTrack GUI, perhaps under the "misc" 
section? 

Thanks again for this, you're a star!

Original comment by ba...@museproject.co.uk on 10 Jan 2015 at 12:41

[GoogleCodeExporter]

Hello Barry,
I'm glad I could help you...

As for building the OSC output to the Linuxtrack GUI, it is unnecessary - I'll 
try to put together a small GUI (like Wiimote server) that will do that. I'll 
keep you posted...

Kind regards,

Michal

Original comment by f.jo...@email.cz on 10 Jan 2015 at 6:57

[GoogleCodeExporter]

Hello Barry,

I sent you a mail, if it didn't show up, please check your spam folder (some 
providers do blacklist mine provider for some reason)...

Kind regards,

Michal

Original comment by f.jo...@email.cz on 12 Jan 2015 at 7:33

[GoogleCodeExporter]

Hello Matt,

I just uploaded the new test package: 
http://linuxtrack.eu/test/linuxtrack-0.99.12abs-64.zip

It is universal package and it contains the following additions... In ltr_gui, 
on the fourth pane there is a field called focal length - you tweak it, until 
the computed 3D point locations correspond to the real ones. To get the 3D 
point locations, specify the following env. variable before running ltr_gui: 

LINUXTRACK_DBG=3

It will print the points to the terminal, when you run ltr_gui and start 
tracking.

The computed absolute pose is provided as raw data via the advanced interface 
(see the Linuxtrack interface topic in ltr_gui's help and the 
linuxtrack_hello_world_adv.c found in the /opt/linu.../share/linuxtrack 
directory).

Should you encounter any problems, don't hesitate to chime in...

Kind regards,

Michal

Original comment by f.jo...@email.cz on 12 Jan 2015 at 7:45

[GoogleCodeExporter]

Michal,

I had a chance to test the binary you provided, and I can say the values 
printed on the console look quite reasonable! Thanks!

Can you elaborate on retrieving the new absolute points using 
linuxtrack_get_pose_full? I see the linuxtrack_pose_t structure hasn't changed. 
Are the values returned in raw_* the new absolute values?

Is your solution general enough that this feature might make it's way into 
general distribution? Let me know if I can help in any way.

Best,
Matt

P.S. Barry -- sorry to have hijacked your thread!

Original comment by matthew....@gmail.com on 15 Jan 2015 at 10:54

[GoogleCodeExporter]

Hello Matt,
you are right, raw_* contain the absolute values... To get the idea, look at 
the share/linuxtrack/linuxtrack_hello_world_adv.c - just grab the raw_* 
values...

I'm thinking of adding it to the structure somehow - maybe I'll add it to the 
end of the structure and add there the structure length to get sort of 
backwards compatibility... Then I'll be able to add it to the normal package.

Kind regards,

Michal

Original comment by f.jo...@email.cz on 15 Jan 2015 at 1:10

[GoogleCodeExporter]

Michal,

Would it be possible to access the source code for your absolute position mod? 
I'm interested in testing my project on mac and windows hosts. I'd love to 
contribute back somehow -- if you need any help integrating the mod into the 
distributed source I'd be happy to take a look.

Best,
Matt

Original comment by matthew....@gmail.com on 24 Jan 2015 at 9:53

[GoogleCodeExporter]

Hello Matt,

you can find it here: 
https://code.google.com/p/linux-track/source/browse/#svn%2Fbranches%2Fabs

You can normally check it out and compile it as usual... 

Kind regards,

Michal

Original comment by f.jo...@email.cz on 25 Jan 2015 at 12:00

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hi Michal,

just a quick message to let you know about a couple of issues with this. 
Firstly the command line tools does;t work with your latest version of Linux 
Track, I'm still having to use 99.10. Also just wondering if you still planned 
to implement a GUI for it like the Wiimote server? 

All best wishes, 

Barry

Original comment by ba...@museproject.co.uk on 8 Jun 2015 at 3:56

[GoogleCodeExporter]

Hello Barry,
thank you for getting back to me - I was not aware of the problems and I'm 
going to look at that...

I'm still planning that GUI, I was just busy with too many other things...

Kind regards,

Michal

Original comment by f.jo...@email.cz on 8 Jun 2015 at 6:43

[GoogleCodeExporter]

Hi Michal, that's awesome thanks! One other issue I have found with this, if 
the TrackIR receives too many points of light (i.e if you point it at a window 
or any other massive source of IR light) it seems to block up the OSC output 
and data stops getting sent out. 

Hope that feedback helps! :)

Original comment by ba...@museproject.co.uk on 9 Jun 2015 at 7:38

[GoogleCodeExporter]

Hello Barry,
osc_client (cmdline) is now added as a standard part of Linuxtrack delivery.
Packege can be found here: http://linuxtrack.eu/test/linuxtrack-0.99.12fix.dmg

I also changed it a bit, so now you get new pose immediately when it is 
computed.
I'll take a look at the GUI stuff as soon as time permits.

As for the issue you see - Linuxtrack reports only valid pose; when there is 
not enough points to form the pose, you won't get an update. But usually when 
there are more points, it takes three topmost points and tries to compute the 
pose. I can see if I can provide at least the points, but beware that is this 
scenario the tracking is extremely unreliable...

Kind regards,

Michal

Original comment by f.jo...@email.cz on 10 Jun 2015 at 9:11

[GoogleCodeExporter]

Hi Michal, thanks for this. When I say that it blocks up the OSC messages, I 
mean that it stops sending messages full stop. After you hit the TrackIR with 
loads of possible points (by pointing it at sunlight) it refuses to send any 
data out afterwards, even if you reduce the points to just 1. 

Best wishes, 

Original comment by ba...@museproject.co.uk on 11 Jun 2015 at 8:33