search

BigRedT / info-ground

Learning phrase grounding from captioned images through InfoNCE bound on mutual information
http://tanmaygupta.info/info-ground/
Other
72 stars 16 forks source link

Experiment #1

Open alibabadoufu opened 4 years ago

alibabadoufu commented 4 years ago

Hi, thanks for the cool work!

I am wondering if you could construct the negative captions with another alternative:

  1. collecting all the noun words in the training set
  2. randomly substituting the noun word for a pair of image-caption sample with the collection. (remove those synonyms by re-ranking for training)

I am trying to guess how important is the context in this case.

BigRedT commented 4 years ago

Hi @alibabadoufu,

That's a great experiment suggestion and is certainly worth trying! Just one thing to note is that this "random-substitution" baseline while not involving context in substitute sampling, will still involve context during featurization (since we use a language model to compute contextualized word representations). But I agree, this is still a good test to see how important is context during sampling.

You can actually try this easily by modifying sample_neg_bert.py file in case I don't get a chance to run this. I would be very curious to know the results if you do try this!

Thanks, Tanmay