arrenglover
commented
1 year ago @ARTICLE{Delbrucksgoalie,
AUTHOR={Delbruck, Tobi and Lang, Manuel},
TITLE={Robotic goalie with 3 ms reaction time at 4% CPU load using event-based dynamic vision sensor},
JOURNAL={Frontiers in Neuroscience},
VOLUME={7},
YEAR={2013},
URL={https://www.frontiersin.org/articles/10.3389/fnins.2013.00223},
DOI={10.3389/fnins.2013.00223},
ISSN={1662-453X}}https://www.frontiersin.org/articles/10.3389/fnins.2013.00223/full
Assumes objects are clusters
Visual tracking is one of the earliest applications showcasing event-camera potential, in particular Delbruck's Robot Goalie~\cite{Delbrucksgoalie} and Conradt's pencil balancer~\cite{5457625}. Since then there have been a growing interest, and body of work, on event-driven tracking. We are interested in the problem of continuous estimation of an object state relative to the camera, in which both the camera and object move in a textured and cluttered environment.
Tracking the camera's motion itself receives much attention and development of novel algorithms and methods~\cite{EVO, recent_slam?}. The task overlaps with object tracking; i.e. time evolving state estimation, and therefore inspiration for event processing can be drawn from the topic. However, as object tracking has (by definition) dynamic environment and requires segmentation of the target object, algorithms need to be adapted to the target domain.
Much work also exists in tracking visual features, such as corners[] or learned features[]. Such features correspond to a set of spatially continuous patterns, instead of a complex or piecewise object appearance. Features can also be aliased across objects and it is also expected that features adapt and disappear over time. The literature has shown strong position priors are effective tracking assumptions that take advantage of event-camera characteristics (e.g. dense, continuous trails), which should also hold true for object tracking.
A particular feature that naturally arises from event data, and has been used for object tracking, is that of clusters. Clusters have been proposed as tracking objects by parts[], but more often also as representing an entire object as a single cluster[Barranco2018, PiatkowskaPeopleTracking]. The latter works best if it can be guaranteed that objects are always spatially separate and do not traverse in front of other texture in the scene, which it typically untrue for moving camera scenarios.
Finally, generic object segmentation can be performed with simultaneous camera and object motion based on a difference between background and object motion dynamics~\cite{MitrokhinsSegmentationMotion}. Such methods assume that the object is always in motion, with periods of constant velocity. In conditions in which the robot is moving the camera to match speeds with the object, this assumption doesn't hold as the relative speed between camera and object tends to zero, and due to robot control, has inconsistent velocities.
The above methods of camera motion tracking, cluster tracking, and velocity-vector-based segmentation are all unsuitable for the proposed application domain. Therefore we are interested in the particular object tracking sub-domain in which the target is known a-priori, and contraints about relative motion between camera, object, and environment are eliminated. In this sub-domain we have shown our prior work on tracking a circle (i.e. a ball) in position and size~\cite{gloversPF}, but is not designed to generalise well to other objects. Since then other works that can possibly track a particular shape have consisted of employing off-the-shelf computer vision algorithms on custom event representations~\cite{ChensATSLTD}, which uses a ``EdgeBoxes'' detection. Additionally trained networks~\cite{ZhangsHybridBBDetection, ChensATSLTDwithNetwork, ZhangsSpikingAffineEstimator} for object tracking have the ability to be selective to particular objects in the training set.
However, the works in~\cite{ChensATSLTD, ChensATSLTDwithNetwork, ZhangsSpikingAffineEstimator} works still only consider camera motion against a static environment (in which the tracked objects are static), in this scenario the relative contrast between the object and the background remains constant, making the object appearance also constant. In a task in which the object moves relative to the background, the contrast strength, and hence events produced changes over time making it more difficult for tracking. In addition, if the camera is closed-loop tracking the object, the appearance of the object changes drastically as the relative motion between the camera and object goes to zero (and hence few events are produced).
In addition, all of these works it isn't clear if the systems can run online and in real-time. Even if computation times are mentioned, until actual closed-loop robotics task are performed pipeline bottlenecks often go unnoticed, for example pre-processing the event-stream may add significant latency for fast camera motion.