About Observers in the paper

alexa / dialoglue

DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue

https://evalai.cloudcv.org/web/challenges/challenge-page/708/overview

Apache License 2.0

279 stars 25 forks source link

About Observers in the paper #15

Open shunyuzh opened 3 years ago

shunyuzh commented 3 years ago

Hi Mehri, Thanks for your awesome work 'Example-Driven Intent Prediction with Observers', and your open sourcing codebase. How did you add observers to bert model in your codebase? I can't find what is related to [OBS]. Did you use the [PAD] as the [OBS]? And how did you make Observers the tokens that are not attended to? Looking forward to your reply.

shunyuzh commented 3 years ago

@mihail-amazon Hi, could you help?

Shikib commented 2 years ago

Apologies for the extremely late reply. I'm not an official collaborator on this repo, so I did not get a notification about your issue. To answer your question, yes [PAD] tokens were used as [OBS]. By default, [PAD] tokens attend to all other tokens however they are masked by all other tokens. This line in the code is responsible for averaging over the necessary number of observers: https://github.com/alexa/dialoglue/blob/6836630dd6b0f40f755ce974f1628b1db5f0c50f/bert_models.py#L110

shunyuzh commented 2 years ago

Got it! I have also noticed it. Thank you.