麻烦博主帮忙看一下，英文数据训练问题

[Atom-J]

使用英文数据集训练出现如下错误，看了好久不知道怎么解决： Traceback (most recent call last): File "run.py", line 42, in trainer.train() File "F:\Files\lm_ner-main\process\Trainer.py", line 220, in train loss = self.step(bs) File "F:\Files\lm_ner-main\process\Trainer.py", line 248, in step outputs = self.model(input_ids, labels=labels, attention_mask=attention_mask) # File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "F:\biyesheji\lm_ner-main\model\transformer_crf.py", line 52, in forward loss = -1 self.crf(emissions = logits, tags=labels, mask=attention_mask) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "F:\Files\lm_ner-main\model\layers\crf.py", line 100, in forward numerator = self._compute_score(emissions, tags, mask) File "F:\Files\lm_ner-main\model\layers\crf.py", line 213, in _compute_score last_tags = tags[seq_ends, torch.arange(batch_size)] IndexError: index 1065 is out of bounds for dimension 0 with size 256

[Atom-J]

还有一个问题，如果使用英文数据集的话是每个单词一个标签还是每个字母一个标签啊

[wzzzd]

还有一个问题，如果使用英文数据集的话是每个单词一个标签还是每个字母一个标签啊

一个单词一个标签的

[wzzzd]

使用英文数据集训练出现如下错误，看了好久不知道怎么解决： Traceback (most recent call last): File "run.py", line 42, in trainer.train() File "F:\Files\lm_ner-main\process\Trainer.py", line 220, in train loss = self.step(bs) File "F:\Files\lm_ner-main\process\Trainer.py", line 248, in step outputs = self.model(input_ids, labels=labels, attention_mask=attention_mask) # File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "F:\biyesheji\lm_ner-main\model\transformer_crf.py", line 52, in forward loss = -1 self.crf(emissions = logits, tags=labels, mask=attention_mask) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "F:\Files\lm_ner-main\model\layers\crf.py", line 100, in forward numerator = self._compute_score(emissions, tags, mask) File "F:\Files\lm_ner-main\model\layers\crf.py", line 213, in _compute_score last_tags = tags[seq_ends, torch.arange(batch_size)] IndexError: index 1065 is out of bounds for dimension 0 with size 256

看着像句子长度超过Config.py文件里面配置的max_seq_length的长度了

[Atom-J]

使用英文数据集训练出现如下错误，看了好久不知道怎么解决： Traceback (most recent call last): File "run.py", line 42, in trainer.train() File "F:\Files\lm_ner-main\process\Trainer.py", line 220, in train loss = self.step(bs) File "F:\Files\lm_ner-main\process\Trainer.py", line 248, in step outputs = self.model(input_ids, labels=labels, attention_mask=attention_mask) # File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "F:\biyesheji\lm_ner-main\model\transformer_crf.py", line 52, in forward loss = -1 self.crf(emissions = logits, tags=labels, mask=attention_mask) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "F:\Files\lm_ner-main\model\layers\crf.py", line 100, in forward numerator = self._compute_score(emissions, tags, mask) File "F:\Files\lm_ner-main\model\layers\crf.py", line 213, in _compute_score last_tags = tags[seq_ends, torch.arange(batch_size)] IndexError: index 1065 is out of bounds for dimension 0 with size 256

看着像句子长度超过Config.py文件里面配置的max_seq_length的长度了

非常感谢您的回答！

[Atom-J]

使用英文数据集训练出现如下错误，看了好久不知道怎么解决： Traceback (most recent call last): File "run.py", line 42, in trainer.train() File "F:\Files\lm_ner-main\process\Trainer.py", line 220, in train loss = self.step(bs) File "F:\Files\lm_ner-main\process\Trainer.py", line 248, in step outputs = self.model(input_ids, labels=labels, attention_mask=attention_mask) # File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "F:\biyesheji\lm_ner-main\model\transformer_crf.py", line 52, in forward loss = -1 self.crf(emissions = logits, tags=labels, mask=attention_mask) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "F:\Files\lm_ner-main\model\layers\crf.py", line 100, in forward numerator = self._compute_score(emissions, tags, mask) File "F:\Files\lm_ner-main\model\layers\crf.py", line 213, in _compute_score last_tags = tags[seq_ends, torch.arange(batch_size)] IndexError: index 1065 is out of bounds for dimension 0 with size 256

看着像句子长度超过Config.py文件里面配置的max_seq_length的长度了

但是我的数据集里面没有句子的长度超过512，并且我尝试使用conll2003的数据集后报了同样的错误 数据集样例： EU B-ORG rejects O German B-MISC call O to O boycott O British B-MISC lamb O . O

Peter B-PER Blackburn I-PER

BRUSSELS B-LOC 1996-08-22 O

The O European B-ORG Commission I-ORG said O on O Thursday O it O disagreed O with O German B-MISC advice O to O consumers O to O shun O British B-MISC lamb O until O scientists O determine O whether O mad O cow O disease O can O be O transmitted O to O sheep O . O 关于这个错误您是否还有其他思路？感谢

[wzzzd]

使用英文数据集训练出现如下错误，看了好久不知道怎么解决： Traceback (most recent call last): File "run.py", line 42, in trainer.train() File "F:\Files\lm_ner-main\process\Trainer.py", line 220, in train loss = self.step(bs) File "F:\Files\lm_ner-main\process\Trainer.py", line 248, in step outputs = self.model(input_ids, labels=labels, attention_mask=attention_mask) # File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "F:\biyesheji\lm_ner-main\model\transformer_crf.py", line 52, in forward loss = -1 self.crf(emissions = logits, tags=labels, mask=attention_mask) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "F:\Files\lm_ner-main\model\layers\crf.py", line 100, in forward numerator = self._compute_score(emissions, tags, mask) File "F:\Files\lm_ner-main\model\layers\crf.py", line 213, in _compute_score last_tags = tags[seq_ends, torch.arange(batch_size)] IndexError: index 1065 is out of bounds for dimension 0 with size 256

看着像句子长度超过Config.py文件里面配置的max_seq_length的长度了

但是我的数据集里面没有句子的长度超过512，并且我尝试使用conll2003的数据集后报了同样的错误 数据集样例： EU B-ORG rejects O German B-MISC call O to O boycott O British B-MISC lamb O . O

Peter B-PER Blackburn I-PER

BRUSSELS B-LOC 1996-08-22 O

The O European B-ORG Commission I-ORG said O on O Thursday O it O disagreed O with O German B-MISC advice O to O consumers O to O shun O British B-MISC lamb O until O scientists O determine O whether O mad O cow O disease O can O be O transmitted O to O sheep O . O 关于这个错误您是否还有其他思路？感谢

数据处理的逻辑，有改动过哪些地方么？

[Atom-J]

使用英文数据集训练出现如下错误，看了好久不知道怎么解决： Traceback (most recent call last): File "run.py", line 42, in trainer.train() File "F:\Files\lm_ner-main\process\Trainer.py", line 220, in train loss = self.step(bs) File "F:\Files\lm_ner-main\process\Trainer.py", line 248, in step outputs = self.model(input_ids, labels=labels, attention_mask=attention_mask) # File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "F:\biyesheji\lm_ner-main\model\transformer_crf.py", line 52, in forward loss = -1 self.crf(emissions = logits, tags=labels, mask=attention_mask) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "F:\software_of_computer_learning\Anaconda\install_path\envs\Bert_BiLstm\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "F:\Files\lm_ner-main\model\layers\crf.py", line 100, in forward numerator = self._compute_score(emissions, tags, mask) File "F:\Files\lm_ner-main\model\layers\crf.py", line 213, in _compute_score last_tags = tags[seq_ends, torch.arange(batch_size)] IndexError: index 1065 is out of bounds for dimension 0 with size 256

看着像句子长度超过Config.py文件里面配置的max_seq_length的长度了

但是我的数据集里面没有句子的长度超过512，并且我尝试使用conll2003的数据集后报了同样的错误 数据集样例： EU B-ORG rejects O German B-MISC call O to O boycott O British B-MISC lamb O . O Peter B-PER Blackburn I-PER BRUSSELS B-LOC 1996-08-22 O The O European B-ORG Commission I-ORG said O on O Thursday O it O disagreed O with O German B-MISC advice O to O consumers O to O shun O British B-MISC lamb O until O scientists O determine O whether O mad O cow O disease O can O be O transmitted O to O sheep O . O 关于这个错误您是否还有其他思路？感谢

数据处理的逻辑，有改动过哪些地方么？ 感谢您的提醒，我之前确实对DataManager文件做了修改，改回后已经可以顺利开始训练，再次感谢您的耐心回复

[wzzzd]

不客气 ;)