Vertex Identifying when inference time

[GuSangmo]

Hi Kyle, thank you for such a nice paper! I really learned a lot from your work.

Currently, I am trying to tune your model for inference in ASD task (for random video, with no annotation about any bbox or entity)

As mentioned in #4 , this code assumes face bbox and audio-visual features be detected by other models. (in this case, your model used 2DResNet with TSM for feature extraction)

I understood SPELL works on pre-built graph data, where vertex is identified with (video_id, entity_id, timestamp). Could you give me some advice on how to get entity_id when inference?

I tried to integrate L186 ~ L223 of data_loader.py into inference loop as you mentioned, but it seems to require entity_id for node. I thought adding some tracker would help, but I am curious if modifying some of your codes might help this.

Thank you so much, Sangmo

[kylemin]

Hi Sangmo,

Yes, you can use a tracking algorithm to assign the same entity_id to all the face-crops of each person. I think you can refer to Ego4D's starter code to see how it tracks a face-crop across the frames. It uses a short-term tracking algorithm to link face-crops with the same entity_id. There are many real-time algorithms for this type of short-term tracking.

I hope this helps.

Thank you, Kyle

[GuSangmo]

I really appreciate your tips, Kyle! Could you answer me one more question?

I tried to extract feature maps with STE(resnet18-tsm-aug) weight you gave, but I couldn't replicate the accurate result like yours.

I used AudioVideoDatasetAuxLossesForwardPhase(..., clip_length = 11, target_size = (144,144)) , resnet18_two_streams_forward(rgb_stack_size = 11) from models_stage1_tsm.py and STE_forward.py.

I thought data loading technique can be little different with your TSM inference code, since using the original STE_forward.py can make the same data for the first few samples.

# This way dataset[0] and dataset[2] has same video & audio data
# Because of the loading logic by ASC requires padding
dataset = AudioVideoDatasetAuxLossesForwardPhase(...)
audio_data, video_data, .... =  dataset[idx]

Was there any slight modifiation to data loading, or STE_forward.py? (Your bbox feature concatenation part seems to be in SPELL logic part, I guess)

It would be a big help if I could know there was something magic or my mistake. Otherwise I could work with the given pretrained weight.

Best regards, Sangmo

[hugoobui]

Hello @GuSangmo, I'm also working on making this code for real-time purposes. Did you manage to do it ?

[GuSangmo]

@hugoobui Sorry for late reply. I couldn't do it, so I chose the other model(LightASD).

In the case of GNN-postprocessing models(e.g. EASEE, SPELL), I thought bbox-tracking should be done in advance, to build the graph for postprocessing. And I couldn't reproduce those feature-encoder part ;-)

My task was due 23 August, so there can be some improvements in this domain. LightASD is much faster, but I also thought additional engineering is necessary( i.e. wait for 5 frames to be encoded to the model)! I hope you could tackle this problem. Best regards, Sangmo