search

OFA-Sys / OFA

Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
Apache License 2.0
2.38k stars 248 forks source link

Information leakage occurs in snli-ve training #357

Open mactavish91 opened 1 year ago

mactavish91 commented 1 year ago

The main difference between VE and TE task is, the premise in TE in a natural language sentence P~text~, instead of an image premise P~image~. However, in OFA experiment setting, the image premise, text premise and text hypothesis are all fed to the encoder, resulting in performance far exceeding other models.

JustinLin610 commented 1 year ago

The related setup of this task is mentioned in the paper.

yingfhu commented 1 year ago

Any comments on this issue? The text premise should not be used in VE task.