NoviScl
commented
3 years ago Hi,
This codebase was using an outdated version of huggingface transformers, I'd recommend that you check the documentation for the new version, which includes example scripts on how to finetune on multi-choice MRC datasets.
First, thank you for sharing the code. I have the following bugs when I train with the dream dataset: 05/24/2021 19:26:24 - INFO - modeling_utils - loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/xlnet-large-cased-pytorch_model.bin from cache at C:\Users\1.cache\torch\pytorchtransformers\distributed-1\f633f4587594591f65958ee58746db3181da87e4110b3d21a13a8cc98a0a22ab.db8dc8babedbb75a56c36fca3e02b016e19fd682e79fb1a928e03c2df977cace Traceback (most recent call last): File "run_xlnet_dream.py", line 693, in
main()
File "run_xlnet_dream.py", line 468, in main
model = XLNetForSequenceClassification.from_pretrained(args.xlnet_model,
File "E:\语义计算与知识检索\project\others\XLNet_DREAM-master\modeling_utils.py", line 403, in from_pretrained
model = cls(config)
File "E:\语义计算与知识检索\project\others\XLNet_DREAM-master\modeling_xlnet.py", line 1122, in init
self.transformer = XLNetModel(config)
File "E:\语义计算与知识检索\project\others\XLNet_DREAM-master\modeling_xlnet.py", line 737, in init
self.word_embedding = nn.Embedding(config.n_token, config.d_model)
File "D:\anaconda\envs\torch\lib\site-packages\torch\nn\modules\sparse.py", line 109, in init
self.weight = Parameter(torch.Tensor(num_embeddings, embedding_dim))
RuntimeError: Trying to create tensor with negative dimension -1: [-1, 1024]
The training command used is: python run_xlnet_dream.py --data_dir=data --xlnet_model=xlnet-large-cased --output_dir=xlnet_dream --max_seq_length=256 --do_train --do_eval --train_batch_size=32 --eval_batch_size=1 --learning_rate=1e-5 --num_train_epochs=4 --gradient_accumulation_steps=32
The torch version is 1.7.1