Tracking

[qbonnard]

Commercial message: improve the processing rate by 1000% !

Technical message: integrated opencv's tracking feature into Chilitags. A call to the "track()" method can replace one to find(). It is a lot faster (4ms with one tag on a 1600x1200px image with a laptop processor, versus ~35ms), but can not detect new or disappeared tags.

Tracking can also improve a little the detection, for tags that are moving to fast for find(). Such improvement can be toggled with setFindAndTrack()

It can be used in several ways:

• speed up detection when expected tags are known: call track() when the expected tags are found, find() otherwise
• improve responsiveness (at the expense of accuracy) by calling find() after every n calls to track()
• improve responsiveness by calling find() in a thread, and track in another (not tested yet)
• speedup detection by calling find() only in areas of the input image that have changed significantly, and track otherwise

Short comings:

• Not propagated into Chilitags3D yet
• This includes a sample, but no test

This PR is based on the performance tuning PR: https://github.com/chili-epfl/chilitags/pull/37

[ayberkozgur]

This is very good news indeed. I'll test these as soon as I can (next week).

[qbonnard]

Cool ! FYI, the "tracking" performance themselves on this commit are very poor, because of the poorly chosen GROWTH_RATIO constant in Track.cpp. Should you try this commit before I fiddle with it, you may want to set it to 20./10. (aka 2 ;)) for example.

[severin-lemaignan]

Cool! Thanks for this important contribution!

I think we can improve the API, though. From my "stupid end-user" perspective, I do not want to have to choose between find and track (because, from an outsider perspective, the differences/tradeoffs between both are really not obvious). I think we should find a way to intelligently 'merge' them.

What about something like:

• in ROBUST and FAST, we do detection + tracking (as you currently propose for ROBUST only: if it's really not costly, I would have it for FAST as well).
• in FASTEST, I would detect only every X frames (with X=30 for instance -> ~1 per sec), and only use tracking in the meantime.
• I would then remove track from the API

[severin-lemaignan]

Btw, this pull request addresses issue #10

[qbonnard]

OK, and now, for the big fight ;)

I'm not sure about hiding track from the user... the tracking/detecting ratio could very well be a performance setting (as you explained), but there are other use cases that it would not cover:

• if you are expecting only a given subset of tags (e.g. only tag 0), you can call find until 0 is returned, then track as long as it is returned, and back to find if it is lost
• if you want to run find in one thread, and track in another

[qbonnard]

Ooookay, here is a commit merging track into find. It's cleaner, even internally actually. It still allow the user to control whether to track or detect, but it also makes it easy to do it automatically (e.g. always both, or periodically every n input frames). The next step would be to run detection on a part of the image, to have both a constant and faster processing time.

I wasn't very inspired for the variables/method names, so feel free to propose better (I know you like it @severin-lemaignan ;)

[qbonnard]

Implemented the review.

I left DETECT_ONLY, because even if it makes a small difference on performance, this difference can count in some cases, and more importantly, there are cases where we want to disable tracking (our test suite for example: tracking would mean that it's trying to find tags in unrelated still images)

TRACK_AND_PARTIAL_DETECT will be the one that runs the detection only on a sub part of the image. It is not implemented yet, I removed it for now

Last issue remaining: setDefaultDetectionTrigger and AUTO vs DEFAULT The point of exposing setDefaultDetectionTrigger is if the user wants to use a performance preset, and then change only the tracking parameter without setting everything else manually and without passing the wanted detection trigger as a parameter to find() or estimate(). It would be important if estimate didn't pass the parameter to find, but it does, so if you want, I can remove setDefaultDetectionTrigger and sleep well.

[qbonnard]

@severin-lemaignan I removed setDefaultTracking... so is it OK to merge ? (if so, I can do it, or if you want to, please avoid this HORRIBLE history splitting "Merge pull request")

[severin-lemaignan]

I've no time now for in-depth testing, so I trust you on this one! :-) or maybe @ayberkozgur wants to have a look first?

[ayberkozgur]

I have a report deadline tomorrow so I can have a look at it as soon as sometime tomorrow night.

[qbonnard]

Woooow. So far I tried with the detect-live sample, and it looked fine. But on the estimate3d one, its catastrophic. Hum...

[qbonnard]

Oook, that looks better now. The problem was that Refine messed up the order of the corners of the Quads, which was fine as long as a detection was run after wards... which is not the case anymore.

[qbonnard]

Hum... I tried with the javascript version, it's not very convincing, probably because the framerate is too slow to take profit of the tracking. In the end I'm not so sure it's a good idea to activate tracking by default. I'm very tempted to do "DETECT_ONLY" by default (and this way we maintain backward compatibility), in all performance presets. This way we let the users specify something else in the optional parameter of find() and estimate() if they want...

[lekevicius]

I had great success using TRACK_AND_DETECT in my iOS app. Amazing improvement, thank you for your project!

[qbonnard]

Wow thanks @lekevicius ! It's always good to know that people outside Chili use chilitags, especially in not yet merged branches.

It's also interesting that chilitags run on iOS (we have OpenCV to thank for that). How was the build process? Is it reusable or can we integrated it into the repository? How are the performances ? What's the framerate at which resolution ?

[ayberkozgur]

Hey, I just tested the sample and tracking works amazing, good job!

It's robust against partial occlusions, fast motion and deformation. I think it's indeed safe to do what you did, setting TRACK_PERIODICALLY as the default method in the FAST and FASTEST.

I had two issues:

1) Tracking seems too aggressive at points, namely there are moments when a tag leaves the scene and high contrast stuff in its former place are still marked as that tag. I think this could be prevented to some degree by killing the tracking when the tracked "thing" becomes too deformed to be considered a tag. For example, we might consider killing tracking of all concave quads. This will indeed also hurt the deformation robustness.

2) Partial occlusions by the frame border kill tracking immediately. I don't know the details but I'm guessing that this could be expensive to solve. We might consider just leaving this feature out.

[severin-lemaignan]

Ooook! So, no major complains, the code look good to me! My only real question is this one, but I'm likely just wrong.

One concern I have (but this will no prevent the merge of the pull-request) is testing: we do not test chilitags works with tracking. To fix that, we need to develop video-based tests. One 'easy' way to do that is to shoot videos where a set of tag remain always in the field of view of the camera (but possible blurry, etc). We can then simply count on how many frames the tags were not all detected. I'll open a new issue to track video testing.

[ayberkozgur]

I added JNI bindings for tracking as a PR to this branch. Is it OK for everyone?

[lekevicius]

Some comments from a me, a random dude.

I had sub-par performance of my use case with chilitags without using tracking. I was building an interactive table not unlike Reactable, so enter / exit "events" of tags was the most important thing. Without tracking, tags that were being moved (a very common case) would flicker, creating unreliable stream of events.

I knew that I was missing tracking, and was super happy to find that somebody made a basic version of that. So I checked out development branch, and after some parameter tweaking my problem was solved!

This slightly answers the general discussion: tracking improves the quality of Chilitags reliability in continuos tracking very much, and would be beneficial to many users. Even if you don't enable it by default, make sure it's a very visible option, and well introduced to anyone discovering the project.

In my use case tracking was a bit aggressive - it would keep tracking non-tag objects if something like it would be in the frame. My suggestion is to try "losing" the tag if it suddenly deforms very much, or overall if very flat, long or in an "unlikely" angle (judging by four corners).

And a final note on performance. I used it in iOS app, on three different models (A5 - iPod Touch, A6 - iPhone 5, and A7 - iPhone 5S). I can no longer do benchmarks, but from the memory, Chilitags would run on 720p video frames (to make sure there is enough resolution for tags from large distance). Processing time on empty frames (without tags) was something like 400 / 250 / 110. So iPhone 5S felt very responsive. With 6-8 tags and tracking it would drop by about 1.5x.

Fun fact: in the end we had to limit to 7 FPS because iPhone 5S would overheat :smile:

Sorry for a long comment. And thank you for this project.

[ayberkozgur]

Wow, glad to hear it actually compiles and runs on iOS, thanks!

Also thanks for sharing your experience on tracking, we're working on robustness against false positives.

[severin-lemaignan]

Is there anything still preventing the merge of this pull-request?

[qbonnard]

Nope, it was on my todo list for the afternoon.

On Tue, Aug 19, 2014 at 1:38 PM, Séverin Lemaignan <notifications@github.com

wrote:

Is there anything still preventing the merge of this pull-request?

— Reply to this email directly or view it on GitHub https://github.com/chili-epfl/chilitags/pull/38#issuecomment-52621247.