Ractiv's TouchPlus 3D Stereo Camera software using TUIO cursor

[touchhope]

Hi Martin,

First a big thank you for your fast response.

It's nice of you to be ok having a closer look on code with a unit we would send you.

I've sent an email to backer Robert Sweetman 3 days ago, who was ok to send his unit to Dr Tagliasacchi before he doubt on if it worth his time.

No answer yet, but with that famous Brexit, as a citizen of Her Majesty The Queen of (Un?)United Kingdom, he must be so sad or so happy, depending on his vote !

Other people were ready to send their (or one of their) unit(s) : 10000 units were manufactured as we can read on Lai's career resume !

I don't know exactly the function of the win_cursor_plus component, but there's a win_cursor_plus_fallback which is launched if :

"if (IsWindows8OrGreater())

    child_module_path = executable_path + slash + "win_cursor_plus" + slash + "win_cursor_plus" + extension0;

else

    child_module_path = executable_path + slash + "win_cursor_plus_fallback" + slash + "win_cursor_plus" + extension0;"

It seems it must receive udp messages of tracking data coming from track_plus process.

I wonder why they choose TUIO csharp cursor instead of TUIO c++ cursor ? Wouldn't it be easier to code everything in c++ to avoid the use of .net framework ?

It seems they've coded win_cursor_plus in csharp to be compliant with csharp TUIO. Maybe at first, c++ TUIO wasn't available to do what they want with cursor ?

All we know is that v0.55b was running a functionnal cursor as seen on Alexandru Bauer's youtube video : https://www.youtube.com/watch?v=ulFTmjiwmAA but noone has the source code for that build.

MJ made a Google drive of the only buggy installer, here : https://drive.google.com/folderview?id=0B06wt0SuwL2PSDFBUk01U1ZWUkk&usp=sharing

So, now talking about the software path, here the escaping characters problem that I diagnostic here on the very last commit in Dec 2015 : https://github.com/Ractiv/touch_plus_source_code/issues/36

Would you have a solution to allow running that "tasklist" command in software path with blank spaces, as it's running directly from build folder without any blank spaces ?

I remember, while testing that last commit, having maybe a TUIO cursor launched from win_cursor_plus process, but obtaining a blue message from windows 10 saying this software couldn't be run in that version of windows and that the manufacturer have had to be contacted. Maybe a manifest file problem ?

And what about hardware z-mapping (https://en.wikipedia.org/wiki/Texture_mapping) ? Maybe Edwin Catmull the pioneer at Pixar could help us ? Maybe Ractiv has lied to us from the moment they've tested the first manufactured units with the software and saw that Etron chipset would never be accessible to any optimisation due to wrong electronic design ?

Is a full software solution possible without full hardware access ?

Is a chessboard image disparity z-mapping system a good option ?

Am I mad of still hoping to at least get a minimal working cursor as seen on Youtube ?

Well it's funny to test all their attempts on funtionnalities through the differents commits, like commit 8b60c27 on 1 Sep 2015 I've just been testing : a medium window showing a moving cloud of points and lines which look like malaxing a spagghetti mass !

What a pity Umar Nizamani's windows home pc as died because he seemed to be confident in solving the csharp process problem : "This part of the code is commented in the code which I was able to run but still wouldn't move the cursor based on my movements. I will see if I can get this working over the weekend so that even with buggy movement end users can have a controllable cursor."

I have desperately tried to recompile the very earlier commit 65abaf6 on 8 Jul 2015 : https://github.com/Ractiv/touch_plus_source_code/issues/39 to maybe help someone solving the cursor problem.

And after all, a good big trackpad would be a good option but autralian backer Lachlan Pollock said : "But having briefly used the Touch+ I've got to say I enjoyed the experience and found it a lot more natural than the leap ever was on a flat surface."

It's good to be a "geek" on technology innovations !

Let's dream on what we can do sharing knowledge !

Cheers,

Christian or Touchhope at http://gharbi.me/ractiv/

[mkalten]

Dear Christian @touchhope , that's a lot of information to digest ... to be honest I don't really know much about this device, but as far as I can tell it seems to be quite abandoned. In order to be able to help you at all I'd need to know to which extent the driver software is actually making use of the TUIO framework. The choice for the .NET implementation seems so far reasonable, if the developers actually use .NET, the C++ implementation isn't any different at all.

Looking at the time and effort that may be needed to fix this, and also considering the overall feature set of your device, I'd say that spending 40EUR for a LEAP motion controller may be the more efficient solution for your problems.

[touchhope]

Hi Martin,

See below : Robert is asking me to give him an address to send his unit to you if you're still ok.

I know you won't probably spent to much time on this sinking project as Dr Tagliasacchi said but some technical advices from you would probably be usefull to get a minimal working cursor.

Cheers,

Christian

Objet : Re: Ractiv's TouchPlus 3D Stereo Camera software using TUIO cursor Date : 24/06/16 23:11 De : "Robert Sweetman" robertiansweetman@gmail.com A : "christian leriche" leriche.christian@sfr.fr Cc : "Lokki" lokkip@hotmail.com, "Vic Jang" vic.jang@gmail.com, martin.kaltenbrunner@jku.at, "wassgha" wassgha@gmail.com

Hi Christian

Yep - this thing is still sitting on a shelf so just give me an address to send it to

Cheers

Robert

On 22 June 2016 at 07:17, christian leriche leriche.christian@sfr.fr wrote:

Hi Robert,

How are you ? Are you for the Brexit or not ? Are you still ok to send your Ractiv's touchplus unit to a university teacher engineer ?

Well, it's because Martin Kaltenbrunner, who is Professor at the Interface Culture Lab at the University of Art and Design in Linz, Austria, have just answered my demand of help :

mkalten commented 12 hours ago • edited

"hi @touchhope, honestly I never heard of this device before ... looks interesting though.
as far as I understand, the manufacturer left you with non-functional drivers and sdk?
from what I saw quickly reviewing their code, they are using the CSHARP TUIO client in their software,
which may be a hint that their driver actually sends touch data encoded as /tuio/2Dcur cursors.

you could try to use one of the available TUIO client demos for any platform or language and see if you can receive anything on their UDP port (which seems to be different from the standard TUIO UDP port 3333)

you could also try to update the TUIO CSHARP code, which would also be the right place for this discussion, since this here is the C++ version ... https://github.com/mkalten/TUIO11_NET

if you want to provide me with such a device, I can have a closer look of course

edit: it also seems that they modified/extended the TUIO protocol, which probably results in making it incompatible with standard TUIO software. what's exactly the function of the win_cursor_plus component?"

Could you tell me soon if you would be still ok, so that we could see that with Dr. Martin Kaltenbrunner ?

Cheers,

Christian (or touchhope at http://gharbi.me/ractiv/)

[touchhope]

http://reactivision.sourceforge.net/

Absolutely interesting !

[touchhope]

A Fast and Robust Fingertips Tracking Algorithm for Vision-Based Multi-touch Interaction :

https://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=9&ved=0ahUKEwjHqu2Xj9XNAhVGhywKHX4ZDs0QFghbMAg&url=http%3A%2F%2Fvc.cs.nthu.edu.tw%2Fhome%2Fpaper%2Fcodfiles%2Fphwang%2F201405231142%2FA%2520Fast%2520and%2520Robust%2520Fingertips%2520Tracking%2520Algorithm%2520for%2520Vision-Based%2520Multi-touch%2520Interaction.pptx&usg=AFQjCNGbYXgHuTMu1QB8EpGk9l8aeERraA&bvm=bv.126130881,d.bGg&cad=rja

[touchhope]

Hi the touchhopers !

After testing a lot of commits, the most interesting to me is Regular update d679834 on 24 Jul 2015 https://github.com/Ractiv/touch_plus_source_code/commit/d6798349bd6b4d385c4bef170a068a914b99287f because it shows us how win_cursor_plus process is supposed to control TUIO cursor.

Indeed, when you compile this commit (a lot of dependencies are missing in its dependencies folder and must be completed with other dependencies comming from other later commits) you can get a medium window called "image_visualization_pointer_mapper" showing us the supposed behavior of TUIO cursor as seen on Alexandru Bauer's Youtube on v0.55b release without full available source code on github.

2 circles get superimposed as finger is coming to a centered small pointing area and distance to get in contact with virtual surface on keyboard is detected by pointer mapper system which is called from main.cpp in track_plus in this commit.

Unfortunately this commit isn't stable enough and you get crashing of deamon_plus and also track_plus process in few minutes.

But no TUIO cursor is launched through win_cursor_plus process and indeed nothing can be listened using https://github.com/gregharding/TUIOListener !

However I've set TUIO port to standard 3333 port instead of random TUIO port.

How could we send to TUIO cursor tracking data sent to window called "image_visualization_pointer_mapper" ?

How could it had worked on Alexandru Bauer pc as seen on Youtube : https://www.youtube.com/watch?v=ulFTmjiwmAA ?

To get a very stable commit without the tasklist command problem in software path, you can try the last commit before "processes.cpp stability fixes" commit 6b7e36e on 2 Nov 2015, "In progress: new algorithm for finger segmentation" https://github.com/Ractiv/touch_plus_source_code/commit/266d5e64dcdf8a0e19e44aa64059843017303f72.

The problem is that 4 points keyboard keys location calibration system seems to have been abandoned in the last commits (new circle initialization calibrating jauge) and we have to analyze how we can reconstruct logical pointer mapper system.

Indeed the notion of "palm point" seems to be important to inject tracking data to the win_cursor_plus process under csharp TUIO protocol.

So "Improved location of palm_point" on commit 2c0207a on 4 Nov 2015, post "processes.cpp stability fixes" commit must also be examined to try to understand how pointing circles get controled in "image_visualization_pointer_mapper" window https://github.com/Ractiv/touch_plus_source_code/commit/2c0207a2e0ddfe1e0ec9fd43dcca81f714f5b0fc.

At worst, we could restart from the beginning from Umar Nizamani work and programming the proposed mathematical solution "A Fast and Robust Fingertips Tracking Algorithm for Vision-Based Multi-touch Interaction" by Qunqun Xie, Guoyuan Liang, Cheng Tang, and Xinyu Wu, and maybe mixing it with Matthew H. Fogle csharp proposal to get interaction the TUIO cursor !

So we need enraged newbies programmers to get it working with the minimal cursor function because it seems coding experts won't lose their precious time on that damn project !

What a pity that Lai Schwe isn't kind enough to share minimal information so that we must made a sort of retro-engineering to try to understand something making lot of tests !

Cheers !

Your desperate Touchhope still trying to find something interesting... until the hope will definitely died !

[touchhope]

http://www.fastcodesign.com/3061282/microsofts-plan-to-build-the-ultimate-gestural-ui

https://blogs.microsoft.com/next/2016/06/26/talking-hands-microsoft-researchers-moving-beyond-keyboard-mouse/#sm.0000dl3f5izjwcx3snd2l8n3indhl

[touchhope]

http://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html

http://docs.opencv.org/2.4/doc/tutorials/features2d/feature_detection/feature_detection.html