search

allenai / bilm-tf

Tensorflow implementation of contextualized word representations from bi-directional language models
Apache License 2.0
1.62k stars 452 forks source link

How could I use word-level ELMO easily? #212

Open ricky9123 opened 5 years ago

ricky9123 commented 5 years ago

It's easy to train a word-level ELMO by this repo, and I could remove char_cnn parameter to achieve this.

However, when I used this ELMO in allennlp, I found that it doesn't support word-level ELMO because it will always find the char_cnn parameter.

Did I have something wrong?

ricky9123 commented 5 years ago

Some key logs are following:

  File "/users4/bcwang/miniconda2/envs/py3/lib/python3.6/site-packages/allennlp/modules/token_embedders/elmo_token_embedder.py", line 128, in from_params
    scalar_mix_parameters=scalar_mix_parameters)
  File "/users4/bcwang/miniconda2/envs/py3/lib/python3.6/site-packages/allennlp/modules/token_embedders/elmo_token_embedder.py", line 65, in __init__
    scalar_mix_parameters=scalar_mix_parameters)
  File "/users4/bcwang/miniconda2/envs/py3/lib/python3.6/site-packages/allennlp/modules/elmo.py", line 110, in __init__
    vocab_to_cache=vocab_to_cache)
  File "/users4/bcwang/miniconda2/envs/py3/lib/python3.6/site-packages/allennlp/modules/elmo.py", line 524, in __init__
    self._token_embedder = _ElmoCharacterEncoder(options_file, weight_file, requires_grad=requires_grad)
  File "/users4/bcwang/miniconda2/envs/py3/lib/python3.6/site-packages/allennlp/modules/elmo.py", line 310, in __init__
    self._load_weights()
  File "/users4/bcwang/miniconda2/envs/py3/lib/python3.6/site-packages/allennlp/modules/elmo.py", line 398, in _load_weights
    self._load_char_embedding()
  File "/users4/bcwang/miniconda2/envs/py3/lib/python3.6/site-packages/allennlp/modules/elmo.py", line 405, in _load_char_embedding
    char_embed_weights = fin['char_embed'][...]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/users4/bcwang/miniconda2/envs/py3/lib/python3.6/site-packages/h5py/_hl/group.py", line 262, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'char_embed' doesn't exist)"
ricky9123 commented 5 years ago

And my ELMO config is:

{"bidirectional": true, "dropout": 0.1, "lstm": {"cell_clip": 3, "dim": 4096, "n_layers": 2, "proj_clip": 3, "projection_dim": 512, "use_skip_connections": true}, "all_clip_norm_val": 10.0, "n_epochs": 10, "n_train_tokens": 21150, "batch_size": 128, "n_tokens_vocab": 21150, "unroll_steps": 20, "n_negative_samples_batch": 8192}
Pydataman commented 5 years ago

allennlp use both word and char embedding