Associating original detections from a track

[ruffsl]

I'd like to be able to associate original detections from a track in order to then process those observations for classification purposes, e.g. recognise a tracked individual by appearance and assign/update the tracking ID to label or identity. In the README.md#L60, something quite equivalent to affect is aluded to:

In case of detection-to-track fusion (currently not implemented), it is still advisable to publish a CompositeDetectedPerson message (via CompositeDetectedPersons) for each set of detections associated with a track, such that later on it is possible to go back to the original detections from a track, and lookup associated image bounding boxes etc. via the associated detection_id.

Currently, the spencer_tracking_msgs/TrackedPerson message type only includes one detection_id field. From what I traced so far, the detection_id is derived from the last observation in the array of observations used during its initialisation, or updated using only the observation from the best pairing:

1. newTrack->observation = observations.back();
   • while initialising new tracks from candidates
2. 'track->observation = bestPairing->observation;'
   • update track's observation from the best pairing

3. trackedPerson.detection_id = track->observation->id;

   • while publishing tracks

   Additionally, timing information from sensor input to processed output is not preserved, as detections with different modalities are fused, resulting in ill-defined timestamps to publish fused detections under. This also makes message capture/synchronization (say for post processing ROIs) using ROS topic filters via timing not as suitable. See simplified node graph for dataflow for tracking_on_bagfile.launch (click to expand)

   

My question is how could we determine to what track a given observation may have been paired with? Specifically, my goal is to monitor "unlabeled" tracks, find upper body ROI pairings with those given tracks, search and recognise faces from those corresponding ROIs, then "label" those track IDs to identities as their faces come into view.

Thanks for any guidance.

[tlind]

Hi,

if you are using the standard detection-to-detection fusion pipeline, the detection_id in the TrackedPerson message will contain the ID of the fused detection. To find out which original detections have been fused together, you have to subscribe to the detected_persons_composite topic, and keep a memory of those fused detections until the output from the tracker is available.

So instead of synchronizing via timestamp, you buffer the CompositeDetectedPersons until you are able to process them (maybe in a cyclic buffer so that you don't run out of memory). As the assigned (composite) detection IDs are monotonically increasing, detection IDs from different frames won't collide with each other.

I once wrote a Python module for this purpose, but I haven't tested it in a long while so I don't know if it still works. But maybe this can serve as an inspiration:

https://github.com/spencer-project/spencer_people_tracking/blob/master/detection/spencer_detected_person_association/src/spencer_detected_person_association/__init__.py

There is a usage example for the TrackSynchronizer class starting from line 172:

def callback(trackAssociation, humanAttributes):
    # Do something useful with the human attributes message...
    # Use trackAssociation.lookupTrackId(detectionId) to look up the track associated with a particular detection (may be None)

human_attribute_sub = message_filters.Subscriber('/spencer/perception_internal/human_attributes/rgbd_front_top', HumanAttributes)
track_sync = spencer_detected_person_association.TrackSynchronizer(human_attribute_sub, 100)
track_sync.registerCallback(callback)

Edit: I think you at least have to change the topic it subscribes to from fused_detected_persons to detected_persons_composite, as we renamed the message type and topic at some point.

[ruffsl]

Alright, so we've used the TrackSynchronizer and are encountering some steady state synchronization issues. Here is our current approach: main.py

We are using the TrackSynchronizer with the upper body detections then rebroadcasting the detection and track_id correspondence to a new message topic. In this file, we are using the ApproximateTimeSynchronizer to filter the RGB image, the upper body detections, the corresponding ROIs, as well as the track associations mentioned previously.

The current issues we are encountering is that the callback will proceed as expected, publishing image crops with matching track_id overlay, but only for limited time after startup; around 10 seconds. Thereafter the callback encounters issues where the list of raw detections and track associations share no common detection ids.

We hypothesize this maybe due to the fact that the phase shift in publication timestamps between the original detections message and the track association topic we create grows beyond the period of the common publication rate, as such that the ApproximateTimeFilter is aligning messages that do not truly correspond with each other.

Perhaps, you could take a look at our approach and recommend any modifications?