search

nyu-mll / jiant-v1-legacy

The jiant toolkit for general-purpose text understanding models
MIT License
21 stars 9 forks source link

Evaluating on MNLI-diagnostic alone fails #224

Open jeswan opened 4 years ago

jeswan commented 4 years ago

Issue by sleepinyourhat Monday Jul 23, 2018 at 19:39 GMT Originally opened as https://github.com/nyu-mll/jiant/issues/224

eval_tasks = mnli-diagnostic, train_tasks = none ... AttributeError: 'MultiTaskModel' object has no attribute 'mnli_mdl'

If anyone sees easy logic to add to make sure that we create an MNLI model when this task is requested, would be worth adding.

jeswan commented 4 years ago

Comment by sleepinyourhat Monday Jul 23, 2018 at 19:40 GMT

Relevant: #202

jeswan commented 4 years ago

Comment by Jan21 Monday Jul 23, 2018 at 20:12 GMT

I thought that this holds for all tasks which use a classifier from a different task. But now Najoung told me that she is using classifier form another task without explicitly loading that task. I don't see any easy fix now...Will look at it later.

jeswan commented 4 years ago

Comment by pitrack Tuesday Jul 24, 2018 at 22:21 GMT

what's the current way to get around this issue? something like eval_tasks = mnli-diagnostic, train_tasks = mnli, do_train=0, train_for_eval=0?

(curious because i had some time and maybe could work on this)

jeswan commented 4 years ago

Comment by sleepinyourhat Tuesday Jul 24, 2018 at 22:51 GMT

That should work, yes. On Tue, Jul 24, 2018 at 6:21 PM Patrick Xia notifications@github.com wrote:

what's the current way to get around this issue? something like eval_tasks = mnli-diagnostic, train_tasks = mnli, do_train=0, train_for_eval=0?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jsalt18-sentence-repl/jiant/issues/224#issuecomment-407571042, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOZWfiQPXJuMxUi19Lg7IyFTjpghhTQks5uJ53agaJpZM4Vbk9O .

jeswan commented 4 years ago

Comment by sleepinyourhat Friday Jul 27, 2018 at 19:38 GMT

@Jan21 - Did this get fixed?

jeswan commented 4 years ago

Comment by Jan21 Friday Jul 27, 2018 at 20:38 GMT

I don't think so...I will try to understand how the use_classifier logic works and fix it.

jeswan commented 4 years ago

Comment by sleepinyourhat Friday Jul 27, 2018 at 20:39 GMT

No real need, just checking.

jeswan commented 4 years ago

Comment by sleepinyourhat Wednesday Apr 24, 2019 at 22:20 GMT

@zphang Mind taking a look at this, since you're working with related code?

jeswan commented 4 years ago

Comment by zphang Friday Apr 26, 2019 at 16:51 GMT

Looked into this.

It looks like one issue is that to build the corresponding task module requires instantiating the Task object, which requires loading the corresponding task data (that we don't need) along the way. If we're fine with this compromise, this is an easy fix. Otherwise, we need to add functionality to building out the module without the Task object, or building a skeleton Task object without loading the data.

Relatedly, there is some similar logic in https://github.com/nyu-mll/jiant/blob/master/src/models.py#L411-L419 for generating tasks based on the use_classifier overrides, which also highlights the issue of reloading MNLI data.

jeswan commented 4 years ago

Comment by sleepinyourhat Friday Apr 26, 2019 at 17:01 GMT

Hrm—this could turn out to be a problem in the future for very data-rich tasks like MLM. How easy would it be to have an option in the task object init code to prevent any data from being loaded?