Approx-CAL -> CAL fails in ActivityNet&Charades

[escorciav]

Description: unknown error of Approx-CAL -> CAL in ActivityNet-Captions with real queries.

Details: Query at 1153 breaks Approx-CAL -> CAL. The variable moment_indices is an empty list. Thus, the second search is done over no moments 😕.

Dev environment: fast-retrieval branch with stash "approx-cal+cal debug activitynet" at tyler.

[escorciav]

Test on Charades-STA/DiDeMo also failed with a different error.

• Incorrect indexing of videos indices.
• There is a mismatch in the shape of proposals and video_indices attributes of the engine.

[escorciav]

• Move stash into commit.

• Retrieval engine works for DiDeMo, but raises error in description for both Charades-STA and ActivityNet-Captions.

• Observations

  The number of clips without linked moments on Charades-STA was more than half the total number of clips. It's better to validate this again.

• Moved hypothesis here:

  1. the video might only have one clip, and we have the constraint of at least two clips per moment.

  2. Maybe padding in dataset enabled, #24 . Thus, there are fake clips per video for which no moment is linked to.

• Next

  1. Estimate/Compute number of clips where a moment can't be associated with.

  2. Check padding

• Questions, concerns

  1. Did we observe this with TwoStageClipPlusMCNFast ?

[escorciav]

• We confirmed that we are adding "spurious" clips to the database.

• By enabling padding in the UntrimmedSMCN dataset, we significantly reduce the number of clips where a moment can't be associated with. From 60% to 5% rate.

  Shouldn't this number should be zero? we may need to investigate this further in another issue.

It sounds that focusing on fixing #24, it's the way to go.