Closed attardi closed 4 years ago
Is it OK on master?
Same problem with master brand on English
with transformers 2.1.1
Epoch 1 / 50000: train: Loss: 3.5856 UAS: 45.99% LAS: 21.43% dev: Loss: 3.5484 UAS: 49.89% LAS: 23.37% ... result = self.forward(*input, **kwargs) File "/homenfs/tempGPU/iwpt2020/.env/lib64/python3.6/site-packages/transformers/modeling_bert.py", line 212, in forward attention_scores = attention_scores / math.sqrt(self.attention_head_size) RuntimeError: CUDA out of memory. Tried to allocate 2.10 GiB (GPU 0; 14.76 GiB total capacity; 5.76 GiB already allocated; 1.64 GiB free; 12.33 GiB reserved in total by PyTorch)
with transformers 2.10.0, it happens an epoch later:
Epoch 2 / 50000: train: Loss: 2.9198 UAS: 48.00% LAS: 30.77% dev: Loss: 2.9559 UAS: 50.39% LAS: 33.26% Traceback (most recent call last): ... result = self.forward(*input, *kwargs) File "/homenfs/tempGPU/iwpt2020/.env/lib64/python3.6/site-packages/transformers/modeling_bert.py", line 314, in forward hidden_states = self.LayerNorm(hidden_states + input_tensor) File "/homenfs/tempGPU/iwpt2020/.env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 550, in call result = self.forward(input, **kwargs) File "/homenfs/tempGPU/iwpt2020/.env/lib64/python3.6/site-packages/transformers/modeling_bert.py", line 235, in forward return outputs RuntimeError: CUDA out of memory. Tried to allocate 2.23 GiB (GPU 0; 14.76 GiB total capacity; 6.34 GiB already allocated; 1.59 GiB free; 12.38 GiB reserved in total by PyTorch)
Could you print the model and fields. This may be due to too many labels.
----------------+--------------------------
Param | Value
----------------+--------------------------
bert_model | bert-base-cased
n_embed | 100
n_char_embed | 50
n_feat_embed | 100
n_bert_layers | 4
embed_dropout | 0.33
n_lstm_hidden | 400
n_lstm_layers | 3
lstm_dropout | 0.33
n_mlp_arc | 500
n_mlp_rel | 100
mlp_dropout | 0.33
lr | 0.002
mu | 0.9
nu | 0.9
epsilon | 1e-12
clip | 5.0
decay | 0.75
decay_steps | 5000
batch_size | 5000
epochs | 50000
patience | 100
min_freq | 2
fix_len | 20
mode | train
punct | False
ftrain | ../train-dev/UD_English-EWT/en_ewt-ud-train.conllu
fdev | ../train-dev/UD_English-EWT/en_ewt-ud-dev.conllu
ftest | ../test-turkunlp/en2.conllu
fembed |
unk | [unk]
conf | config.ini
file | exp/en_ewt-bert
preprocess | True
device | cuda
seed | 1
threads | 16
tree | False
proj | False
feat | bert
buckets | 32
fields | exp/en_ewt-bert/fields
model | exp/en_ewt-bert/model
n_words | 8867
n_feats | 28996
n_rels | 50
pad_index | 0
unk_index | 1
bos_index | 2
feat_pad_index | 0
----------------+--------------------------
----------------+-------------------------- Param | Value
----------------+-------------------------- bert_model | bert-base-cased
n_embed | 100
n_char_embed | 50
n_feat_embed | 100
n_bert_layers | 4
embed_dropout | 0.33
n_lstm_hidden | 400
n_lstm_layers | 3
lstm_dropout | 0.33
n_mlp_arc | 500
n_mlp_rel | 100
mlp_dropout | 0.33
lr | 0.002
mu | 0.9
nu | 0.9
epsilon | 1e-12
clip | 5.0
decay | 0.75
decay_steps | 5000
batch_size | 5000
epochs | 50000
patience | 100
min_freq | 2
fix_len | 20
mode | train
punct | False
ftrain | ../train-dev/UD_English-EWT/en_ewt-ud-train.conllu fdev | ../train-dev/UD_English-EWT/en_ewt-ud-dev.conllu ftest | ../test-turkunlp/en2.conllu fembed |
unk | [unk]
conf | config.ini
file | exp/en_ewt-bert
preprocess | True
device | cuda
seed | 1
threads | 16
tree | False
proj | False
feat | bert
buckets | 32
fields | exp/en_ewt-bert/fields
model | exp/en_ewt-bert/model
n_words | 8867
n_feats | 28996
n_rels | 50
pad_index | 0
unk_index | 1
bos_index | 2
feat_pad_index | 0
----------------+--------------------------
Sorry for being vague, I mean
print(self.model)
print(self.fields)
Model( (word_embed): Embedding(8867, 100) (feat_embed): BertEmbedding(n_layers=4, n_out=100, pad_index=0) (embed_dropout): IndependentDropout(p=0.33) (lstm): BiLSTM(200, 400, num_layers=3, dropout=0.33) (lstm_dropout): SharedDropout(p=0.33, batch_first=True) (mlp_arc_d): MLP(n_in=800, n_out=500, dropout=0.33) (mlp_arc_h): MLP(n_in=800, n_out=500, dropout=0.33) (mlp_rel_d): MLP(n_in=800, n_out=100, dropout=0.33) (mlp_rel_h): MLP(n_in=800, n_out=100, dropout=0.33) (arc_attn): Biaffine(n_in=500, n_out=1, bias_x=True) (rel_attn): Biaffine(n_in=100, n_out=50, bias_x=True, bias_y=True) (criterion): CrossEntropyLoss() )
CoNLL(ID=None, FORM=((words): Field(pad=
BTW. Why do you distinguish:
if args.feat in ('char', 'bert'):
self.fields = CoNLL(FORM=(self.WORD, self.FEAT),
HEAD=self.ARC, DEPREL=self.REL)
else:
self.fields = CoNLL(FORM=self.WORD, CPOS=self.FEAT,
HEAD=self.ARC, DEPREL=self.REL)
and then do:
if args.feat in ('char', 'bert'):
self.WORD, self.FEAT = self.fields.FORM
else:
self.WORD, self.FEAT = self.fields.FORM, self.fields.CPOS
You might just use:
self.fields = CoNLL(FORM=(self.WORD, self.FEAT),
HEAD=self.ARC, DEPREL=self.REL)
and then
self.WORD, self.FEAT = self.fields.FORM
Since POS tags and words are in different columns, I have to make a distinction.
I see. Dozat uses both word and POS feature. You replace POS with BERT. Have you tried keeping POS?
Yeah, --feat=tag
can make model behaves as described in the paper.
I mean using both POS and BERT.
Yes, I have tried this way, but under the strong baseline of BERT, POS tags have very little effect.
Nevertheless, my implementation (especially IndependentDropout
) can well support the combination of three or more features with minor modifications.
I agree that BERT is a much richer representation, but it might be worth a try. I also wonder why you use also embeddings for words, besides BERT. Isn't BERT enough?
Since BERT is a character/subword level features, intuitively, word embedding can be a useful supplement. But I did not verify this.
You can make BERT into word features by averaging the workpieces.
You can make BERT into word features by averaging the workpieces.
That's exactly what the current code implements.
Then why do you need both? You might use subwords embeddings only when they are present, as a surrogate of POS, saving memory.
To some extent, for API consistency :joy:.
I am looking for ways to save memory, since I still have that problem with many treebanks. I had been able to train all the UD treebanks with the master branch. Something is causing an increase in memory.
Could you send me the data to reproduce this error?
You can download the data from here: http://ufal.mff.cuni.cz/~zeman/soubory/iwpt2020-train-dev.tgz
I am using bert_model = 'bert-base-multilingual-cased' for languages other than English.
Hi, the issue is caused by some weird sentence annotations like
# sent_id = email-enronsent15_01-0049
# text = >----------------------------------------------------------------------------| | | >----------------------------------------------------------------------------|
1 >----------------------------------------------------------------------------| >----------------------------------------------------------------------------| SYM NFP _ 4 punct 4:punct _
2 | | SYM NFP _ 4 punct 4:punct _
3 | | SYM NFP _ 4 punct 4:punct _
4 >----------------------------------------------------------------------------| >----------------------------------------------------------------------------| SYM NFP _ 0 root 0:root _
# newdoc id = newsgroup-groups.google.com_hiddennook_04d8cc994875d454_ENG_20050207_012300
# sent_id = newsgroup-groups.google.com_hiddennook_04d8cc994875d454_ENG_20050207_012300-0001
# text = [http://www.newsday.com/news/opinion/ny-vpnasa054135614feb05,0,5979821.story?coll=ny-editorials-headlines]
1 [ [ PUNCT -LRB- _ 2 punct 2:punct SpaceAfter=No
2 http://www.newsday.com/news/opinion/ny-vpnasa054135614feb05,0,5979821.story?coll=ny-editorials-headlines http://www.newsday.com/news/opinion/ny-vpnasa054135614feb05,0,5979821.story?coll=ny-editorials-headlines X ADD _ 0 root 0:root SpaceAfter=No
3 ] ] PUNCT -RRB- _ 2 punct 2:punct _
which leads to very large word pieces. You can fix this issue by limiting the length of word pieces:
self.FEAT = SubwordField('bert',
pad=tokenizer.pad_token,
unk=tokenizer.unk_token,
bos=tokenizer.cls_token,
fix_len=args.fix_len,
tokenize=tokenizer.tokenize)
Great, it works on English! I will test it on all treebanks.
I am testing the dev branch, using transformers 2.10.0, somewhat successfully.
However it runs out of CUDA memory on the UD_English-EWT treebank:
Epoch 8 / 50000: Traceback (most recent call last): File "run.py", line 61, in
cmd(args)
File "/homenfs/tempGPU/iwpt2020/parser/parser/cmds/train.py", line 81, in call
loss, train_metric = self.train(train.loader)
File "/homenfs/tempGPU/iwpt2020/parser/parser/cmds/cmd.py", line 91, in train
s_arc, s_rel = self.model(words, feats)
File "/homenfs/tempGPU/iwpt2020/.env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, *kwargs)
File "/homenfs/tempGPU/iwpt2020/parser/parser/model.py", line 92, in forward
feat_embed = self.feat_embed(feats)
File "/homenfs/tempGPU/iwpt2020/.env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(input, **kwargs)
File "/homenfs/tempGPU/iwpt2020/parser/parser/modules/bert.py", line 59, in forward
embed = embed.maskedscatter(mask.unsqueeze(-1), bert[bert_mask])
RuntimeError: CUDA out of memory. Tried to allocate 2.02 GiB (GPU 0; 14.76 GiB total capacity; 6.04 GiB already allocated; 1.06 GiB free; 12.90 GiB reserved in total by PyTorch)
I thought it was because the treebank is large, but there are larger treebanks on which it works:
I tried increasing the buckets to 48, but that did not help. It works though by decreasing the batch_size to 500.
The problem occurs also with transformers 2.1.1 on the UD_English-EWT.