[CLOSED] Edge probing configs & stats script

nyu-mll / jiant-v1-legacy

The jiant toolkit for general-purpose text understanding models

MIT License

21 stars 9 forks source link

[CLOSED] Edge probing configs & stats script #238

Closed jeswan closed 4 years ago

jeswan commented 4 years ago

Issue by iftenney Tuesday Jul 24, 2018 at 18:07 GMT Originally opened as https://github.com/nyu-mll/jiant/pull/238

Configs & scripts to run edge probing experiments on bare ELMo model, either char CNN layer or full bilSTM model. Uses train mode with 'pass' encoder so we don't learn any new biLSTM params.

Also add a script to compile basic stats on edge probing data.

iftenney included the following code: https://github.com/nyu-mll/jiant/pull/238/commits

jeswan commented 4 years ago

Comment by iftenney Wednesday Jul 25, 2018 at 22:05 GMT

@pitrack : I'll add separate configs / scripts for the concat/multiply pooling. I think we should still run both, but going to leave 'attn' as default for now even though it performs worse in abs terms.

jeswan commented 4 years ago

Comment by iftenney Thursday Jul 26, 2018 at 15:58 GMT

@sleepinyourhat , @W4ngatang : can you take a look at models.py and modules.py here? I made a change to apply @pitrack 's ELMo fix to this case, but want to be 100% sure this doesn't break anything shared since this is pretty critical codepath.

demo.conf works and produces the same numbers as before.

jeswan commented 4 years ago

Comment by W4ngatang Thursday Jul 26, 2018 at 17:22 GMT

These look correct to me, but could you also run demo.conf with the settings:

elmo_chars_only = 1, skip_embs = 0
elmo_chars_only = 1, skip_embs = 1
elmo_chars_only = 0, skip_embs = 0
elmo_chars_only = 0, skip_embs = 1 At least as far as making sure there are no size mismatches?

jeswan commented 4 years ago

Comment by iftenney Thursday Jul 26, 2018 at 20:02 GMT

@W4ngatang : tested, everything runs. Going to merge after the dropout fix ( #254) is in.