search

yangheng95 / PyABSA

Sentiment Analysis, Text Classification, Text Augmentation, Text Adversarial defense, etc.;
https://pyabsa.readthedocs.io
MIT License
955 stars 161 forks source link

IndexError: list index out of range | ATEPC English training on Tshirt dataset #59

Closed hitz02 closed 3 years ago

hitz02 commented 3 years ago

Out-of-range error while training ATEPC model - english on T-shirt dataset.

... config.model = ATEPCModelList.LCFS_ATEPC config.evaluate_begin = 5 config.num_epoch = 6 config.log_step = 100 tshirt = ABSADatasetList.TShirt

aspect_extractor = Trainer(config=config, dataset=tshirt, checkpoint_save_mode=1, auto_device=True )

Traceback - >

TShirt dataset is not found locally, search at https://github.com/yangheng95/ABSADatasets Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight']

7 frames /usr/local/lib/python3.7/dist-packages/pyabsa/functional/trainer/trainer.py in init(self, config, dataset, from_checkpoint, checkpoint_save_mode, auto_device) 92 config.model_path_to_save = None 93 ---> 94 self.train() 95 96 def train(self):

/usr/local/lib/python3.7/dist-packages/pyabsa/functional/trainer/trainer.py in train(self) 103 self.config.seed = s 104 if self.checkpoint_save_mode: --> 105 model_path.append(self.train_func(self.config, self.from_checkpoint, self.logger)) 106 else: 107 # always return the last trained model if dont save trained model

/usr/local/lib/python3.7/dist-packages/pyabsa/core/atepc/training/atepc_trainer.py in train4atepc(opt, from_checkpoint_path, logger) 352 while not trainer: 353 try: --> 354 trainer = Instructor(opt, logger) 355 if from_checkpoint_path: 356 model_path = find_files(from_checkpoint_path, '.model')

/usr/local/lib/python3.7/dist-packages/pyabsa/core/atepc/training/atepc_trainer.py in init(self, opt, logger) 70 len(self.train_examples) / self.opt.batch_size / self.opt.gradient_accumulation_steps) * self.opt.num_epoch 71 train_features = convert_examples_to_features(self.train_examples, self.label_list, self.opt.max_seq_len, ---> 72 self.tokenizer, self.opt) 73 all_spc_input_ids = torch.tensor([f.input_ids_spc for f in train_features], dtype=torch.long) 74 all_input_mask = torch.tensor([f.input_mask for f in train_features], dtype=torch.long)

/usr/local/lib/python3.7/dist-packages/pyabsa/core/atepc/dataset_utils/data_utils_for_training.py in convert_examples_to_features(examples, label_list, max_seq_len, tokenizer, opt) 188 text_right = '' 189 aspect = '' --> 190 prepared_inputs = prepare_input_for_atepc(opt, tokenizer, text_left, text_right, aspect) 191 lcf_cdm_vec = prepared_inputs['lcf_cdm_vec'] 192 lcf_cdw_vec = prepared_inputs['lcf_cdw_vec']

/usr/local/lib/python3.7/dist-packages/pyabsa/core/atepc/dataset_utils/atepc_utils.py in prepare_input_for_atepc(opt, tokenizer, text_left, text_right, aspect) 60 61 if 'lcfs' in opt.model_name or opt.use_syntax_based_SRD: ---> 62 syntacticaldist, = get_syntax_distance(text_raw, aspect, tokenizer, opt) 63 else: 64 syntactical_dist = None

/usr/local/lib/python3.7/dist-packages/pyabsa/core/apc/dataset_utils/apc_utils.py in get_syntax_distance(text_raw, aspect, tokenizer, opt) 240 # the following two functions are both designed to calculate syntax-based distances 241 if opt.srd_alignment: --> 242 syntactical_dist = syntax_distance_alignment(raw_tokens, dist, opt.max_seq_len, tokenizer) 243 else: 244 syntactical_dist = pad_syntax_based_srd(raw_tokens, dist, tokenizer, opt)[1]

/usr/local/lib/python3.7/dist-packages/pyabsa/core/apc/dataset_utils/apc_utils.py in syntax_distance_alignment(tokens, dist, max_seq_len, tokenizer) 38 if bert_tokens != text: 39 while text or bert_tokens: ---> 40 if text[0] == ' ' or text[0] == '\xa0': # bad case handle 41 text = text[1:] 42 dep_dist = dep_dist[1:]

IndexError: list index out of range

yangheng95 commented 3 years ago

TShirt dataset is not found locally, search at https://github.com/yangheng95/ABSADatasets Fail to remove the temp file C:\Users\chuan\AppData\Local\Temp\tmpuwy9h_4m Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.bias']

It works for me, maybe it cause by different encoding in your environment, you can try:

PYTHONIOENCODING=UTF8 python train_atepc.py
hitz02 commented 3 years ago

I am using Google Colab and I checked the default encoding using the below command and its UTF-8

import sys

sys.getdefaultencoding()

Can you check what might be another possible reason?

yangheng95 commented 3 years ago

Did you reclone (update) the datasets?

hitz02 commented 3 years ago

Sorry, can you explain what do you mean by reclone datasets?

I pip installed the updated version of pyabsa and my understanding is the atepc trainer automatically downloads the dataset before the training starts.

Let me know if there is any other approach that I missed out on.

yangheng95 commented 3 years ago

To reclone the datasets, you can delete the downloaded datasets, and the code should download the datasets again

yangheng95 commented 3 years ago

I updated the Tshirt atepc dataset few days ago, but the downloaded dataset can't be redownloaded automaticly.

https://github.com/yangheng95/ABSADatasets/tree/master/datasets/atepc_datasets/TShirt

hitz02 commented 3 years ago

Same error with the updated dataset as well. Can you check using google colab? I am running ATEPC English training using a checkpoint example.

yangheng95 commented 3 years ago

Same error with the updated dataset as well. Can you check using google colab? I am running ATEPC English training using a checkpoint example.

I am sorry for that, I will try on colab, however does this error occur while using every datasets?

hitz02 commented 3 years ago

Checked with Twitter dataset. Same index error.

Traceback -

Twitter dataset is not found locally, search at https://github.com/yangheng95/ABSADatasets Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias']

7 frames /usr/local/lib/python3.7/dist-packages/pyabsa/functional/trainer/trainer.py in init(self, config, dataset, from_checkpoint, checkpoint_save_mode, auto_device) 92 config.model_path_to_save = None 93 ---> 94 self.train() 95 96 def train(self):

/usr/local/lib/python3.7/dist-packages/pyabsa/functional/trainer/trainer.py in train(self) 103 self.config.seed = s 104 if self.checkpoint_save_mode: --> 105 model_path.append(self.train_func(self.config, self.from_checkpoint, self.logger)) 106 else: 107 # always return the last trained model if dont save trained model

/usr/local/lib/python3.7/dist-packages/pyabsa/core/atepc/training/atepc_trainer.py in train4atepc(opt, from_checkpoint_path, logger) 352 while not trainer: 353 try: --> 354 trainer = Instructor(opt, logger) 355 if from_checkpoint_path: 356 model_path = find_files(from_checkpoint_path, '.model')

/usr/local/lib/python3.7/dist-packages/pyabsa/core/atepc/training/atepc_trainer.py in init(self, opt, logger) 70 len(self.train_examples) / self.opt.batch_size / self.opt.gradient_accumulation_steps) * self.opt.num_epoch 71 train_features = convert_examples_to_features(self.train_examples, self.label_list, self.opt.max_seq_len, ---> 72 self.tokenizer, self.opt) 73 all_spc_input_ids = torch.tensor([f.input_ids_spc for f in train_features], dtype=torch.long) 74 all_input_mask = torch.tensor([f.input_mask for f in train_features], dtype=torch.long)

/usr/local/lib/python3.7/dist-packages/pyabsa/core/atepc/dataset_utils/data_utils_for_training.py in convert_examples_to_features(examples, label_list, max_seq_len, tokenizer, opt) 188 text_right = '' 189 aspect = '' --> 190 prepared_inputs = prepare_input_for_atepc(opt, tokenizer, text_left, text_right, aspect) 191 lcf_cdm_vec = prepared_inputs['lcf_cdm_vec'] 192 lcf_cdw_vec = prepared_inputs['lcf_cdw_vec']

/usr/local/lib/python3.7/dist-packages/pyabsa/core/atepc/dataset_utils/atepc_utils.py in prepare_input_for_atepc(opt, tokenizer, text_left, text_right, aspect) 60 61 if 'lcfs' in opt.model_name or opt.use_syntax_based_SRD: ---> 62 syntacticaldist, = get_syntax_distance(text_raw, aspect, tokenizer, opt) 63 else: 64 syntactical_dist = None

/usr/local/lib/python3.7/dist-packages/pyabsa/core/apc/dataset_utils/apc_utils.py in get_syntax_distance(text_raw, aspect, tokenizer, opt) 240 # the following two functions are both designed to calculate syntax-based distances 241 if opt.srd_alignment: --> 242 syntactical_dist = syntax_distance_alignment(raw_tokens, dist, opt.max_seq_len, tokenizer) 243 else: 244 syntactical_dist = pad_syntax_based_srd(raw_tokens, dist, tokenizer, opt)[1]

/usr/local/lib/python3.7/dist-packages/pyabsa/core/apc/dataset_utils/apc_utils.py in syntax_distance_alignment(tokens, dist, max_seq_len, tokenizer) 38 if bert_tokens != text: 39 while text or bert_tokens: ---> 40 if text[0] == ' ' or text[0] == '\xa0': # bad case handle 41 text = text[1:] 42 dep_dist = dep_dist[1:]

IndexError: list index out of range

XuMayi commented 3 years ago

Hello, you can set ”atepc_config_english.srd_alignment = False“ to close the srd_alignment function. This does not have a significant impact on the performance of the model. The SRD in LCFS series models are partly calculated by the SpaCy tool, and the text tokens are split by BERT tokenizer. Considering there are some differences between the BERT tokenizer and SpaCy tokenizer, srd_alignment function is designed to align the SRD and the text tokens. We will modify the srd_alignment function later. Thank you for your feedback.

XuMayi commented 3 years ago

The problem has been solved in the new version of PyABSA.

hitz02 commented 3 years ago

Thanks for resolving the above issue. I am currently trying out training using checkpoints by following this link - https://github.com/yangheng95/PyABSA/blob/release/examples/aspect_term_extraction/train_atepc_based_on_checkpoint.py

While the model is training, it was not able to load the checkpoint. It says the checkpoint is broken. Secondly, it failed to detect Cuda and starting running on CPU on Google Colab. Check below prints, I have highlighted them in bold and italic.

TShirt dataset is not found locally, search at https://github.com/yangheng95/ABSADatasets Invalid device string: '0' Downloading: 100% 28.0/28.0 [00:00<00:00, 621B/s] Downloading: 100% 570/570 [00:00<00:00, 13.0kB/s] Downloading: 100% 232k/232k [00:00<00:00, 644kB/s] Downloading: 100% 466k/466k [00:00<00:00, 622kB/s] Downloading: 100% 440M/440M [00:13<00:00, 33.5MB/s] Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']

yangheng95 commented 3 years ago

Thank you for your carefully report, we are woking on releasing a stable version however many situation are not eassy to consider, e.g., have cuda or not. I got the error and will fix it immediately.

yangheng95 commented 3 years ago

FYI,I assume you can manually avoid the error by setting auto_device='cuda' and choose another checkpoint, because the checkpoint your are loading is misuploaded.

yangheng95 commented 3 years ago

Please try v1.1, it should have fix the cuda detection, I will upload new checkpoints soon

yangheng95 commented 3 years ago

This issue is all set in v1.1, please test and report any problem, thank you very much!

hitz02 commented 3 years ago

Hi, The checkpoint is still broken it seems.

TShirt dataset is not found locally, search at https://github.com/yangheng95/ABSADatasets Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias']

yangheng95 commented 3 years ago

Thanks for your deep report in many situations which really helps us. It is fixed in pathch v1.1.1. Have fun!

hitz02 commented 3 years ago

Hi, I am not sure if this is an error, but the model got trained once and then again started with different batch size.

Please check the logs -

TShirt dataset is not found locally, search at https://github.com/yangheng95/ABSADatasets Downloading: 100% 28.0/28.0 [00:00<00:00, 751B/s] Downloading: 100% 570/570 [00:00<00:00, 19.3kB/s] Downloading: 100% 232k/232k [00:00<00:00, 875kB/s] Downloading: 100% 466k/466k [00:00<00:00, 846kB/s] Downloading: 100% 440M/440M [00:09<00:00, 51.8MB/s] Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias', 'cls.seq_relationship.weight']

This is the aspect extractor aims to extract aspect and predict sentiment, note that use_bert_spc is disabled while extracting aspects and classifying sentiment! Load aspect extractor from training Config used in Training:

model: <class 'pyabsa.core.atepc.models.lcfs_atepc.LCFS_ATEPC'> --> Active optimizer: adamw --> Active learning_rate: 2e-05 --> Active pretrained_bert: bert-base-uncased --> Active use_bert_spc: False --> Active max_seq_len: 80 --> Active SRD: 3 --> Active lcf: cdw --> Active dropout: 0.5 --> Active l2reg: 5e-05 --> Active num_epoch: 5 --> Active batch_size: 16 --> Active seed: 1 --> Active embed_dim: 768 --> Active hidden_dim: 768 --> Active polarities_dim: 3 --> Active log_step: 50 --> Active gradient_accumulation_steps: 1 --> Active dynamic_truncate: True --> Active srd_alignment: True --> Active evaluate_begin: 0 --> Active dataset_file: {'train': ['/content/datasets/atepc_datasets/TShirt/Menstshirt_Train.xml.seg.atepc'], 'test': ['/content/datasets/atepc_datasets/TShirt/Menstshirt_Test_Gold.xml.seg.atepc']} --> Active device: cuda --> Active device_name: Tesla T4 --> Active model_name: lcfs_atepc --> Active Version: 1.1.1 --> Active dataset_path: TShirt --> Active save_mode: 0 --> Active model_path_to_save: None --> Active num_labels: 6 --> Active max_test_metrics: {'max_apc_test_acc': 0, 'max_apc_test_f1': 0, 'max_ate_test_f1': 0} --> Active metrics_of_this_checkpoint: {'apc_acc': 88.09, 'apc_f1': 71.44, 'ate_f1': 78.88} --> Active use_syntax_based_SRD: False --> Default window: lr --> Default initializer: xavieruniform --> Default Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias', 'cls.seq_relationship.weight']

  • This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
  • This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Using bos_token, but it is not set yet. Using eos_token, but it is not set yet. 100%|██████████| 1870/1870 [00:17<00:00, 109.88it/s, convert examples to features] 100%|██████████| 470/470 [00:04<00:00, 109.56it/s, convert examples to features] 2021-08-19 12:17:06,491 INFO: >>> model: <class 'pyabsa.core.atepc.models.lcfs_atepc.LCFS_ATEPC'> --> Active 2021-08-19 12:17:06,492 INFO: >>> optimizer: adamw --> Active 2021-08-19 12:17:06,497 INFO: >>> learning_rate: 2e-05 --> Active 2021-08-19 12:17:06,498 INFO: >>> pretrained_bert: bert-base-uncased --> Active 2021-08-19 12:17:06,500 INFO: >>> use_bert_spc: False --> Active 2021-08-19 12:17:06,502 INFO: >>> max_seq_len: 80 --> Active 2021-08-19 12:17:06,504 INFO: >>> SRD: 3 --> Active 2021-08-19 12:17:06,505 INFO: >>> lcf: cdw --> Active 2021-08-19 12:17:06,508 INFO: >>> dropout: 0.5 --> Active 2021-08-19 12:17:06,509 INFO: >>> l2reg: 5e-05 --> Active 2021-08-19 12:17:06,511 INFO: >>> num_epoch: 5 --> Active 2021-08-19 12:17:06,514 INFO: >>> batch_size: 1 --> Active 2021-08-19 12:17:06,515 INFO: >>> seed: 2 --> Active 2021-08-19 12:17:06,517 INFO: >>> embed_dim: 768 --> Active 2021-08-19 12:17:06,519 INFO: >>> hidden_dim: 768 --> Active 2021-08-19 12:17:06,522 INFO: >>> polarities_dim: 3 --> Active 2021-08-19 12:17:06,524 INFO: >>> log_step: 50 --> Active 2021-08-19 12:17:06,527 INFO: >>> gradient_accumulation_steps: 1 --> Active 2021-08-19 12:17:06,529 INFO: >>> dynamic_truncate: True --> Active 2021-08-19 12:17:06,531 INFO: >>> srd_alignment: True --> Active 2021-08-19 12:17:06,534 INFO: >>> evaluate_begin: 0 --> Active 2021-08-19 12:17:06,536 INFO: >>> dataset_file: {'train': ['/content/datasets/atepc_datasets/TShirt/Menstshirt_Train.xml.seg.atepc'], 'test': ['/content/datasets/atepc_datasets/TShirt/Menstshirt_Test_Gold.xml.seg.atepc']} --> Active 2021-08-19 12:17:06,537 INFO: >>> device: cuda --> Active 2021-08-19 12:17:06,539 INFO: >>> device_name: Tesla T4 --> Active 2021-08-19 12:17:06,541 INFO: >>> model_name: lcfs_atepc --> Active 2021-08-19 12:17:06,544 INFO: >>> Version: 1.1.1 --> Active 2021-08-19 12:17:06,545 INFO: >>> dataset_path: TShirt --> Active 2021-08-19 12:17:06,548 INFO: >>> save_mode: 0 --> Active 2021-08-19 12:17:06,555 INFO: >>> model_path_to_save: None --> Active 2021-08-19 12:17:06,556 INFO: >>> num_labels: 6 --> Active 2021-08-19 12:17:06,557 INFO: >>> max_test_metrics: {'max_apc_test_acc': 0, 'max_apc_test_f1': 0, 'max_ate_test_f1': 0} --> Active 2021-08-19 12:17:06,558 INFO: >>> metrics_of_this_checkpoint: {'apc_acc': 88.09, 'apc_f1': 71.44, 'ate_f1': 78.88} --> Active 2021-08-19 12:17:06,559 INFO: >>> use_syntax_based_SRD: False --> Default 2021-08-19 12:17:06,560 INFO: >>> window: lr --> Default 2021-08-19 12:17:06,561 INFO: >>> initializer: xavieruniform --> Default 2021-08-19 12:17:10,741 INFO: Running training for Aspect Term Extraction 2021-08-19 12:17:10,743 INFO: Num examples = 1870 2021-08-19 12:17:10,748 INFO: Batch size = 1 2021-08-19 12:17:10,750 INFO: Num steps = 9350
yangheng95 commented 3 years ago

Nope, the final checkpoint is loaded and print the training args. but if it does perform another training process, do feel free to contact me.

hitz02 commented 3 years ago

Hi, as I said, it continues to perform another training with different batch size. Check the log shared above. Ideally, it should end with a training summary.

yangheng95 commented 3 years ago

Hi, as I said, it continues to perform another training with different batch size. Check the log shared above. Ideally, it should end with a training summary. The default experiment config take 1, 2, 3 as the seeds, so it will train using all seeds (3 times indeed), but I am not sure why the batch size changed, I am locating the error.

yangheng95 commented 3 years ago

config = ATEPCConfigManager.get_atepc_config_english()

checkpoint_path = ATEPCCheckpointManager.get_checkpoint(checkpoint='english')

config.model = ATEPCModelList.LCFS_ATEPC config.evaluate_begin = 4 config.num_epoch = 5

TShirt = ABSADatasetList.TShirt aspect_extractor = Trainer(config=config, dataset=TShirt, from_checkpoint=checkpoint_path, checkpoint_save_mode=1, auto_device=True )

Can you show me your trainging code? e.g., a script based on any example.

hitz02 commented 3 years ago

Here is my code - 

config = ATEPCConfigManager.get_atepc_config_english()
checkpoint_path = ATEPCCheckpointManager.get_checkpoint(checkpoint='english')

config.model = ATEPCModelList.LCFS_ATEPC
config.evaluate_begin = 0
config.num_epoch = 5
tshirt = ABSADatasetList.TShirt

aspect_extractor = Trainer(config=config,
                           dataset=tshirt,
                           from_checkpoint=checkpoint_path,
                           checkpoint_save_mode=0,
                           auto_device='cuda'
                           )

I have a few follow up questions -

  1. How can we perform batch inference for ATE?
  2. How can we provide a custom directory path to 1) models (BERT and ATEPC checkpoints) and 2) datasets in the code above? My organization has restricted google drive in its environment and hence it fails to autodownload.
  3. Does the current version support distributed training and inference? Or any plans to make it available in the future?
yangheng95 commented 3 years ago

The error is located and being repaired. I release fix patch 1.1.2 to resolve the batch-size changing problem in multiple pieces of training. But if you just need to train once, simply set a single seed.

For your questions:

  1. I have a plan to support tensor-based batch inference, it should come in one of the next versions.
  2. PyABSA uses current work directory-based searching to find models and datasets, you can give the absolute path or dir name for datasets and downloaded checkpoints.
  3. There is no plan to support distributed training and inference since distributed training is kind of dependent on hardware, environment, and demands. I suggest you that forking this repo to add distributed training feature.
hitz02 commented 3 years ago

Hi,

I tried running the trainer by giving checkpoint path and data set path -

config = ATEPCConfigManager.get_atepc_config_english()
config.model = ATEPCModelList.LCFS_ATEPC
config.evaluate_begin = 0
config.num_epoch = 2

aspect_extractor = Trainer(config=config,
                           dataset=</path/to/tshirt_dataset directory>,
                           from_checkpoint=<path/to/english_ATEPC_model directory>,
                           checkpoint_save_mode=0,
                           auto_device='cuda'
                           )

Got below error -

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.seq_relationship.bias']

warning! detected error line(s) in input file:<!DOCTYPE html>

Seems to be ConnectionError, retry in 60 seconds...

I have already downloaded the dataset and model checkpoints and given path to the trainer. What else does it expect and download? My organization has restricted network access and hence the trainer code fails.

Let me know if I have missed anything here.

yangheng95 commented 3 years ago

Hi,

I tried running the trainer by giving checkpoint path and data set path -

config = ATEPCConfigManager.get_atepc_config_english()
config.model = ATEPCModelList.LCFS_ATEPC
config.evaluate_begin = 0
config.num_epoch = 2

aspect_extractor = Trainer(config=config,
                           dataset=</path/to/tshirt_dataset directory>,
                           from_checkpoint=<path/to/english_ATEPC_model directory>,
                           checkpoint_save_mode=0,
                           auto_device='cuda'
                           )

Got below error -

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.seq_relationship.bias']

  • This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
  • This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Using bos_token, but it is not set yet. Using eos_token, but it is not set yet.

warning! detected error line(s) in input file:

Seems to be ConnectionError, retry in 60 seconds...

I have already downloaded the dataset and model checkpoints and given path to the trainer. What else does it expect and download? My organization has restricted network access and hence the trainer code fails.

Let me know if I have missed anything here.

I will review the code. Did you set the dataset path of atepc datasets?

hitz02 commented 3 years ago

Yes, I downloaded the t-shirt dataset to my local and gave its path ( folder with train and test atepc files).

hitz02 commented 3 years ago

Hi, your code example is for performing inference and I am trying out training on tshirt dataset using english checkpoint. The paths given are -

data_path = '/data/hitz02/tshirt_datasets' #this path contains train and test files
checkpoint_path = '/data/hitz02/lcf_atepc_cdw_apcacc_82.77_apcf1_72.41_atef1_80.97/' #this path contains the state dict and other 3 files
yangheng95 commented 3 years ago 
atepc_config_english = ATEPCConfigManager.get_atepc_config_english()
atepc_config_english.num_epoch = 10
atepc_config_english.evaluate_begin = 4
atepc_config_english.lot_step = 100
atepc_config_english.model = ATEPCModelList.LCF_ATEPC
# SemEval = ABSADatasetList.SemEval
dataset_path = r'integrated_datasets/datasets/atepc_datasets/TShirt'
aspect_extractor = ATEPCTrainer(config=atepc_config_english,
                                dataset=dataset_path,
                                from_checkpoint='ATEPC_ENGLISH_TRAINED_MODEL',
                                checkpoint_save_mode=1,
                                auto_device=True
                                )

Sorry I made a mistake, this training script works, too. Can you try this?

hitz02 commented 3 years ago

The ATEPCTrainer parameter -> from_checkpoint='ATEPC_ENGLISH_TRAINED_MODEL'

will require the checkpoint to download from google drive right?

The same checkpoint I have already downloaded and placed on my local. I wish to give the local path of this downloaded checkpoint and then train on tshirt dataset. Can this be done?

yangheng95 commented 3 years ago

ATEPC_ENGLISH_TRAINED_MODEL is a local path of my environment, because I already download them befor, so

from_checkpoint='ATEPC_ENGLISH_TRAINED_MODEL'

cam be replaced by

from_checkpoint='/data/hitz02/lcf_atepc_cdw_apcacc_82.77_apcf1_72.41_atef1_80.97/'

but I recommend to use from_checkpoint=lcf_atepc_cdw_apcacc_82.77_apcf1_72.41_atef1_80.97 as the parameter, the code will search the checkpoint in current working path.

hitz02 commented 3 years ago

Tried from_checkpoint=lcf_atepc_cdw_apcacc_82.77_apcf1_72.41_atef1_80.97

Same error - 

warning! detected error line(s) in input file:<!DOCTYPE html>

Seems to be ConnectionError, retry in 60 seconds...

I think it tries to download something else (apart from data and checkpoint) and fails due to restriction.

yangheng95 commented 3 years ago

I see, this error is probably caused by illegal input, but I cant reporudce your problem.

https://github.com/yangheng95/PyABSA/blob/8fd2a8bcd4bb6f23f25d458a794a716f8fcd9273/pyabsa/core/atepc/dataset_utils/data_utils_for_training.py#L82

I am puzzled by the input <!DOCTYPE html> Can you show me a breif view your dataset

hitz02 commented 3 years ago

I am not using any custom dataset. Just downloaded the Tshirt dataset from your repo and placed on local path and used the data path in the trainer code data_path = '/data/hitz02/tshirt_datasets'

The path contains 2 files - 

Menstshirt_Train.xml.seg.atepc
Menstshirt_Test_Gold.xml.seg.atepc
yangheng95 commented 3 years ago

It's a weired error, indeed. According to my test, 

dataset_path = r'integrated_datasets/datasets/atepc_datasets/TShirt'
aspect_extractor = ATEPCTrainer(config=atepc_config_english,
                                dataset=dataset_path,
                                from_checkpoint='lcf_atepc_cdw_apcacc_80.7_apcf1_68.67_atef1_80.55',
                                checkpoint_save_mode=1,
                                auto_device=True
                                )

may be functional similar to your script, it works. I suppose that the locating of dataset is not correct.

hitz02 commented 3 years ago

Yes, Thanks. The problem was with the dataset that I downloaded. I deleted and downloaded again. And it worked.

How can I change the config to run only with single seed?

yangheng95 commented 3 years ago

e.g., 

atepc_config_english.seed = 1
hitz02 commented 3 years ago

Hi,

This is regarding saving the model. When I keep the checkpoint_save_model=1, it creates lot of intermediate state dict files in GBs. To avoid that, I kept it as checkpoint_save_model=0 Now the model is not saved but stayed as trainer object in memory. How can I save it to disk using the trainer object? Do we have save model method implemented for it? Can we pickle it?

yangheng95 commented 3 years ago

I recommend to set checkpoint_save_model=1, with evaluate_begin=num_epoch-1, and you can set a large threshold evaluate per steps. e.g., log_step=100. This will reduce the intermediate files.

yangheng95 commented 3 years ago

This original probelm has been sovled, so I will close this issue. Please feel free to report new error or raise enquiries.

hitz02 commented 3 years ago

Got keyerror when I tried to save checkpoint

checkpoint_path = 'lcf_atepc_cdw_apcacc_82.77_apcf1_72.41_atef1_80.97'
data_path = '/data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt'

config.model = ATEPCModelList.LCFS_ATEPC
config.evaluate_begin = 5
config.num_epoch = 6
config.log_step = 100
config.seed = 2

aspect_extractor = ATEPCTrainer(config=config,
                           dataset=data_path,
                           from_checkpoint=checkpoint_path,
                           checkpoint_save_mode=1,
                           auto_device='cuda'
                           )

KeyError: 'Fail to load the model from /data/hitz02/checkpoints/lcfs_atepc_cdw_apcacc_89.57_apcf1_72.5_atef1_78.53/! the checkpoint is broken, or maybe the checkpoint is not compatible with this version.'

yangheng95 commented 3 years ago

Hi, What the version are you using? Did you update to 1.1.7?

hitz02 commented 3 years ago

Was using 1.1.5 and updated to 1.1.7 and tried again. Same keyerror encountered.

Traceback -

This is the aspect extractor aims to extract aspect and predict sentiment, note that use_bert_spc is disabled while extracting aspects and classifying sentiment!
Load aspect extractor from /data/hitz02/checkpoints/lcfs_atepc_cdw_apcacc_89.57_apcf1_72.5_atef1_78.53/

TypeError                                 Traceback (most recent call last)
/data/hitz02/env/lib/python3.6/site-packages/pyabsa/core/atepc/prediction/aspect_extractor.py in __init__(self, model_arg, sentiment_map)
     58                 config_path = find_file(model_arg, '.config', exclude_key=['__MACOSX'])
---> 59 
     60                 print('config: {}'.format(config_path))

TypeError: expected str, bytes or os.PathLike object, not NoneType

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-59-d4b7ed1bdcf7> in <module>
      3                            from_checkpoint=checkpoint_path,
      4                            checkpoint_save_mode=1,
----> 5                            auto_device='cuda'
      6                            )

/data/hitz02/env/lib/python3.6/site-packages/pyabsa/functional/trainer/trainer.py in __init__(self, config, dataset, from_checkpoint, checkpoint_save_mode, auto_device)
    107             self.config.seed = [self.config.seed]
    108 
--> 109         model_path = []
    110         seeds = self.config.seed
    111         for _, s in enumerate(seeds):

/data/hitz02/env/lib/python3.6/site-packages/pyabsa/functional/trainer/trainer.py in train(self)
    126 
    127 
--> 128 class APCTrainer(Trainer):
    129     pass
    130 

/data/hitz02/env/lib/python3.6/site-packages/pyabsa/core/atepc/prediction/aspect_extractor.py in __init__(self, model_arg, sentiment_map)
     78                 else:
     79                     self.tokenizer = BertTokenizer.from_pretrained(self.opt.pretrained_bert, do_lower_case=True)
---> 80 
     81                 self.tokenizer.bos_token = self.tokenizer.bos_token if self.tokenizer.bos_token else '[CLS]'
     82                 self.tokenizer.eos_token = self.tokenizer.eos_token if self.tokenizer.eos_token else '[SEP]'

KeyError: 'Fail to load the model from /data/hitz02/checkpoints/lcfs_atepc_cdw_apcacc_89.57_apcf1_72.5_atef1_78.53/! the checkpoint is broken, or maybe the checkpoint is not compatible with this version.'
yangheng95 commented 3 years ago

I see, does the path contain .config and .state_dict files?

hitz02 commented 3 years ago

I realized the model it is trying to load is not created at first place in the checkpoints directory. I have 2 checkpoints with different acc values created but not the one which we see in the traceback. Also both the checkpoints created do have config and state dict files.

yangheng95 commented 3 years ago

image Another weird error, I test the similar code, without errors. Can you share the checkpoint to me via Google Drive or sth to debug?

hitz02 commented 3 years ago

As I said, it is trying to load checkpoint which isn't created. Check the logs and errors below -

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Using bos_token, but it is not set yet.
Using eos_token, but it is not set yet.
100%|██████████| 1870/1870 [00:20<00:00, 93.24it/s, convert examples to features] 
100%|██████████| 470/470 [00:04<00:00, 94.46it/s, convert examples to features] 
2021-08-24 16:55:16,833 INFO: >>> model: <class 'pyabsa.core.atepc.models.lcfs_atepc.LCFS_ATEPC'>  --> Active
INFO:lcfs_atepc:>>> model: <class 'pyabsa.core.atepc.models.lcfs_atepc.LCFS_ATEPC'>  --> Active
2021-08-24 16:55:16,835 INFO: >>> optimizer: adamw  --> Active
INFO:lcfs_atepc:>>> optimizer: adamw  --> Active
2021-08-24 16:55:16,836 INFO: >>> learning_rate: 2e-05  --> Active
INFO:lcfs_atepc:>>> learning_rate: 2e-05  --> Active
2021-08-24 16:55:16,838 INFO: >>> pretrained_bert: bert-base-uncased  --> Active
INFO:lcfs_atepc:>>> pretrained_bert: bert-base-uncased  --> Active
2021-08-24 16:55:16,840 INFO: >>> use_bert_spc: False  --> Active
INFO:lcfs_atepc:>>> use_bert_spc: False  --> Active
2021-08-24 16:55:16,841 INFO: >>> max_seq_len: 80  --> Active
INFO:lcfs_atepc:>>> max_seq_len: 80  --> Active
2021-08-24 16:55:16,843 INFO: >>> SRD: 3  --> Active
INFO:lcfs_atepc:>>> SRD: 3  --> Active
2021-08-24 16:55:16,844 INFO: >>> lcf: cdw  --> Active
INFO:lcfs_atepc:>>> lcf: cdw  --> Active
2021-08-24 16:55:16,845 INFO: >>> dropout: 0.5  --> Active
INFO:lcfs_atepc:>>> dropout: 0.5  --> Active
2021-08-24 16:55:16,846 INFO: >>> l2reg: 5e-05  --> Active
INFO:lcfs_atepc:>>> l2reg: 5e-05  --> Active
2021-08-24 16:55:16,847 INFO: >>> num_epoch: 6  --> Active
INFO:lcfs_atepc:>>> num_epoch: 6  --> Active
2021-08-24 16:55:16,848 INFO: >>> batch_size: 16  --> Active
INFO:lcfs_atepc:>>> batch_size: 16  --> Active
2021-08-24 16:55:16,849 INFO: >>> seed: 2  --> Active
INFO:lcfs_atepc:>>> seed: 2  --> Active
2021-08-24 16:55:16,850 INFO: >>> embed_dim: 768  --> Active
INFO:lcfs_atepc:>>> embed_dim: 768  --> Active
2021-08-24 16:55:16,851 INFO: >>> hidden_dim: 768  --> Active
INFO:lcfs_atepc:>>> hidden_dim: 768  --> Active
2021-08-24 16:55:16,852 INFO: >>> polarities_dim: 3  --> Active
INFO:lcfs_atepc:>>> polarities_dim: 3  --> Active
2021-08-24 16:55:16,853 INFO: >>> log_step: 100  --> Active
INFO:lcfs_atepc:>>> log_step: 100  --> Active
2021-08-24 16:55:16,858 INFO: >>> gradient_accumulation_steps: 1  --> Active
INFO:lcfs_atepc:>>> gradient_accumulation_steps: 1  --> Active
2021-08-24 16:55:16,859 INFO: >>> dynamic_truncate: True  --> Active
INFO:lcfs_atepc:>>> dynamic_truncate: True  --> Active
2021-08-24 16:55:16,860 INFO: >>> srd_alignment: True  --> Active
INFO:lcfs_atepc:>>> srd_alignment: True  --> Active
2021-08-24 16:55:16,861 INFO: >>> evaluate_begin: 5  --> Active
INFO:lcfs_atepc:>>> evaluate_begin: 5  --> Active
2021-08-24 16:55:16,862 INFO: >>> dataset_file: {'train': ['/data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt/Menstshirt_Train.xml.seg.atepc'], 'test': ['/data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt/Menstshirt_Test_Gold.xml.seg.atepc']}  --> Active
INFO:lcfs_atepc:>>> dataset_file: {'train': ['/data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt/Menstshirt_Train.xml.seg.atepc'], 'test': ['/data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt/Menstshirt_Test_Gold.xml.seg.atepc']}  --> Active
2021-08-24 16:55:16,863 INFO: >>> device: cuda  --> Active
INFO:lcfs_atepc:>>> device: cuda  --> Active
2021-08-24 16:55:16,866 INFO: >>> device_name: NVIDIA GeForce GTX 1080 Ti  --> Active
INFO:lcfs_atepc:>>> device_name: NVIDIA GeForce GTX 1080 Ti  --> Active
2021-08-24 16:55:16,867 INFO: >>> model_name: lcfs_atepc  --> Active
INFO:lcfs_atepc:>>> model_name: lcfs_atepc  --> Active
2021-08-24 16:55:16,868 INFO: >>> Version: 1.1.1  --> Active
INFO:lcfs_atepc:>>> Version: 1.1.1  --> Active
2021-08-24 16:55:16,869 INFO: >>> dataset_path: /data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt  --> Active
INFO:lcfs_atepc:>>> dataset_path: /data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt  --> Active
2021-08-24 16:55:16,870 INFO: >>> save_mode: 1  --> Active
INFO:lcfs_atepc:>>> save_mode: 1  --> Active
2021-08-24 16:55:16,872 INFO: >>> model_path_to_save: /data/hitz02/checkpoints  --> Active
INFO:lcfs_atepc:>>> model_path_to_save: /data/hitz02/checkpoints  --> Active
2021-08-24 16:55:16,873 INFO: >>> num_labels: 6  --> Active
INFO:lcfs_atepc:>>> num_labels: 6  --> Active
2021-08-24 16:55:16,874 INFO: >>> max_test_metrics: {'max_apc_test_acc': 90.21, 'max_apc_test_f1': 74.78, 'max_ate_test_f1': 80.75}  --> Active
INFO:lcfs_atepc:>>> max_test_metrics: {'max_apc_test_acc': 90.21, 'max_apc_test_f1': 74.78, 'max_ate_test_f1': 80.75}  --> Active
2021-08-24 16:55:16,875 INFO: >>> metrics_of_this_checkpoint: {'apc_acc': 89.57, 'apc_f1': 72.5, 'ate_f1': 78.53}  --> Active
INFO:lcfs_atepc:>>> metrics_of_this_checkpoint: {'apc_acc': 89.57, 'apc_f1': 72.5, 'ate_f1': 78.53}  --> Active
2021-08-24 16:55:16,876 INFO: >>> use_syntax_based_SRD: False  --> Default
INFO:lcfs_atepc:>>> use_syntax_based_SRD: False  --> Default
2021-08-24 16:55:16,878 INFO: >>> window: lr  --> Default
INFO:lcfs_atepc:>>> window: lr  --> Default
2021-08-24 16:55:16,879 INFO: >>> initializer: xavier_uniform_  --> Default
INFO:lcfs_atepc:>>> initializer: xavier_uniform_  --> Default
2021-08-24 16:55:17,384 INFO: ***** Running training for Aspect Term Extraction *****
INFO:lcfs_atepc:***** Running training for Aspect Term Extraction *****
2021-08-24 16:55:17,386 INFO:   Num examples = 1870
INFO:lcfs_atepc:  Num examples = 1870
2021-08-24 16:55:17,387 INFO:   Batch size = 16
INFO:lcfs_atepc:  Batch size = 16
2021-08-24 16:55:17,389 INFO:   Num steps = 696
INFO:lcfs_atepc:  Num steps = 696
100%|██████████| 117/117 [01:36<00:00,  1.21it/s, Epoch:0 | No evaluation until epoch:5]
100%|██████████| 117/117 [01:36<00:00,  1.21it/s, Epoch:1 | No evaluation until epoch:5]
100%|██████████| 117/117 [01:38<00:00,  1.19it/s, Epoch:2 | No evaluation until epoch:5]
100%|██████████| 117/117 [01:37<00:00,  1.20it/s, Epoch:3 | No evaluation until epoch:5]
100%|██████████| 117/117 [01:37<00:00,  1.20it/s, Epoch:4 | No evaluation until epoch:5]
100%|██████████| 117/117 [01:57<00:00,  1.01s/it, Epoch:5 | loss_apc:0.0626 | loss_ate:0.0921 | APC_ACC: 89.57(max:90.21) | APC_F1: 72.5(max:74.78) | ATE_F1: 78.53(max:80.75)] 
2021-08-24 17:05:21,545 INFO: -------------------------------------Training Summary-------------------------------------

INFO:lcfs_atepc:-------------------------------------Training Summary-------------------------------------
2021-08-24 17:05:21,547 INFO:   Max APC Acc: 90.21000 Max APC F1: 74.78000 Max ATE F1: 80.75000 Accumulated Loss: 371.3037539720535
INFO:lcfs_atepc:  Max APC Acc: 90.21000 Max APC F1: 74.78000 Max ATE F1: 80.75000 Accumulated Loss: 371.3037539720535
2021-08-24 17:05:21,548 INFO: -------------------------------------Training Summary-------------------------------------
INFO:lcfs_atepc:-------------------------------------Training Summary-------------------------------------
Training finished, we hope you can share your checkpoint with everybody, please see: https://github.com/yangheng95/PyABSA#how-to-share-checkpoints-eg-checkpoints-trained-on-your-custom-dataset-with-community
2021-08-24 17:05:21,551 INFO: >>> model: <class 'pyabsa.core.atepc.models.lcfs_atepc.LCFS_ATEPC'>  --> Active
INFO:lcfs_atepc:>>> model: <class 'pyabsa.core.atepc.models.lcfs_atepc.LCFS_ATEPC'>  --> Active
2021-08-24 17:05:21,552 INFO: >>> optimizer: adamw  --> Active
INFO:lcfs_atepc:>>> optimizer: adamw  --> Active
2021-08-24 17:05:21,553 INFO: >>> learning_rate: 2e-05  --> Active
INFO:lcfs_atepc:>>> learning_rate: 2e-05  --> Active
2021-08-24 17:05:21,555 INFO: >>> pretrained_bert: bert-base-uncased  --> Active
INFO:lcfs_atepc:>>> pretrained_bert: bert-base-uncased  --> Active
2021-08-24 17:05:21,557 INFO: >>> use_bert_spc: False  --> Active
INFO:lcfs_atepc:>>> use_bert_spc: False  --> Active
2021-08-24 17:05:21,558 INFO: >>> max_seq_len: 80  --> Active
INFO:lcfs_atepc:>>> max_seq_len: 80  --> Active
2021-08-24 17:05:21,559 INFO: >>> SRD: 3  --> Active
INFO:lcfs_atepc:>>> SRD: 3  --> Active
2021-08-24 17:05:21,560 INFO: >>> lcf: cdw  --> Active
INFO:lcfs_atepc:>>> lcf: cdw  --> Active
2021-08-24 17:05:21,560 INFO: >>> dropout: 0.5  --> Active
INFO:lcfs_atepc:>>> dropout: 0.5  --> Active
2021-08-24 17:05:21,561 INFO: >>> l2reg: 5e-05  --> Active
INFO:lcfs_atepc:>>> l2reg: 5e-05  --> Active
2021-08-24 17:05:21,564 INFO: >>> num_epoch: 6  --> Active
INFO:lcfs_atepc:>>> num_epoch: 6  --> Active
2021-08-24 17:05:21,565 INFO: >>> batch_size: 16  --> Active
INFO:lcfs_atepc:>>> batch_size: 16  --> Active
2021-08-24 17:05:21,566 INFO: >>> seed: 2  --> Active
INFO:lcfs_atepc:>>> seed: 2  --> Active
2021-08-24 17:05:21,566 INFO: >>> embed_dim: 768  --> Active
INFO:lcfs_atepc:>>> embed_dim: 768  --> Active
2021-08-24 17:05:21,568 INFO: >>> hidden_dim: 768  --> Active
INFO:lcfs_atepc:>>> hidden_dim: 768  --> Active
2021-08-24 17:05:21,569 INFO: >>> polarities_dim: 3  --> Active
INFO:lcfs_atepc:>>> polarities_dim: 3  --> Active
2021-08-24 17:05:21,570 INFO: >>> log_step: 100  --> Active
INFO:lcfs_atepc:>>> log_step: 100  --> Active
2021-08-24 17:05:21,572 INFO: >>> gradient_accumulation_steps: 1  --> Active
INFO:lcfs_atepc:>>> gradient_accumulation_steps: 1  --> Active
2021-08-24 17:05:21,573 INFO: >>> dynamic_truncate: True  --> Active
INFO:lcfs_atepc:>>> dynamic_truncate: True  --> Active
2021-08-24 17:05:21,573 INFO: >>> srd_alignment: True  --> Active
INFO:lcfs_atepc:>>> srd_alignment: True  --> Active
2021-08-24 17:05:21,574 INFO: >>> evaluate_begin: 5  --> Active
INFO:lcfs_atepc:>>> evaluate_begin: 5  --> Active
2021-08-24 17:05:21,576 INFO: >>> dataset_file: {'train': ['/data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt/Menstshirt_Train.xml.seg.atepc'], 'test': ['/data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt/Menstshirt_Test_Gold.xml.seg.atepc']}  --> Active
INFO:lcfs_atepc:>>> dataset_file: {'train': ['/data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt/Menstshirt_Train.xml.seg.atepc'], 'test': ['/data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt/Menstshirt_Test_Gold.xml.seg.atepc']}  --> Active
2021-08-24 17:05:21,577 INFO: >>> device: cuda  --> Active
INFO:lcfs_atepc:>>> device: cuda  --> Active
2021-08-24 17:05:21,578 INFO: >>> device_name: NVIDIA GeForce GTX 1080 Ti  --> Active
INFO:lcfs_atepc:>>> device_name: NVIDIA GeForce GTX 1080 Ti  --> Active
2021-08-24 17:05:21,579 INFO: >>> model_name: lcfs_atepc  --> Active
INFO:lcfs_atepc:>>> model_name: lcfs_atepc  --> Active
2021-08-24 17:05:21,580 INFO: >>> Version: 1.1.1  --> Active
INFO:lcfs_atepc:>>> Version: 1.1.1  --> Active
2021-08-24 17:05:21,582 INFO: >>> dataset_path: /data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt  --> Active
INFO:lcfs_atepc:>>> dataset_path: /data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt  --> Active
2021-08-24 17:05:21,583 INFO: >>> save_mode: 1  --> Active
INFO:lcfs_atepc:>>> save_mode: 1  --> Active
2021-08-24 17:05:21,583 INFO: >>> model_path_to_save: /data/hitz02/checkpoints  --> Active
INFO:lcfs_atepc:>>> model_path_to_save: /data/hitz02/checkpoints  --> Active
2021-08-24 17:05:21,584 INFO: >>> num_labels: 6  --> Active
INFO:lcfs_atepc:>>> num_labels: 6  --> Active
2021-08-24 17:05:21,586 INFO: >>> max_test_metrics: {'max_apc_test_acc': 90.21, 'max_apc_test_f1': 74.78, 'max_ate_test_f1': 80.75}  --> Active
INFO:lcfs_atepc:>>> max_test_metrics: {'max_apc_test_acc': 90.21, 'max_apc_test_f1': 74.78, 'max_ate_test_f1': 80.75}  --> Active
2021-08-24 17:05:21,587 INFO: >>> metrics_of_this_checkpoint: {'apc_acc': 89.57, 'apc_f1': 72.5, 'ate_f1': 78.53}  --> Active
INFO:lcfs_atepc:>>> metrics_of_this_checkpoint: {'apc_acc': 89.57, 'apc_f1': 72.5, 'ate_f1': 78.53}  --> Active
2021-08-24 17:05:21,588 INFO: >>> use_syntax_based_SRD: False  --> Default
INFO:lcfs_atepc:>>> use_syntax_based_SRD: False  --> Default
2021-08-24 17:05:21,590 INFO: >>> window: lr  --> Default
INFO:lcfs_atepc:>>> window: lr  --> Default
2021-08-24 17:05:21,590 INFO: >>> initializer: xavier_uniform_  --> Default
INFO:lcfs_atepc:>>> initializer: xavier_uniform_  --> Default
This is the aspect extractor aims to extract aspect and predict sentiment, note that use_bert_spc is disabled while extracting aspects and classifying sentiment!
Load aspect extractor from /data/hitz02/checkpoints/lcfs_atepc_cdw_apcacc_89.57_apcf1_72.5_atef1_78.53/

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/data/hitz02/env/lib/python3.6/site-packages/pyabsa/core/atepc/prediction/aspect_extractor.py in __init__(self, model_arg, sentiment_map)
     58                 config_path = find_file(model_arg, '.config', exclude_key=['__MACOSX'])
---> 59 
     60                 print('config: {}'.format(config_path))

TypeError: expected str, bytes or os.PathLike object, not NoneType

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-59-d4b7ed1bdcf7> in <module>
      3                            from_checkpoint=checkpoint_path,
      4                            checkpoint_save_mode=1,
----> 5                            auto_device='cuda'
      6                            )

/data/hitz02/env/lib/python3.6/site-packages/pyabsa/functional/trainer/trainer.py in __init__(self, config, dataset, from_checkpoint, checkpoint_save_mode, auto_device)
    107             self.config.seed = [self.config.seed]
    108 
--> 109         model_path = []
    110         seeds = self.config.seed
    111         for _, s in enumerate(seeds):

/data/hitz02/env/lib/python3.6/site-packages/pyabsa/functional/trainer/trainer.py in train(self)
    126 
    127 
--> 128 class APCTrainer(Trainer):
    129     pass
    130 

/data/hitz02/env/lib/python3.6/site-packages/pyabsa/core/atepc/prediction/aspect_extractor.py in __init__(self, model_arg, sentiment_map)
     78                 else:
     79                     self.tokenizer = BertTokenizer.from_pretrained(self.opt.pretrained_bert, do_lower_case=True)
---> 80 
     81                 self.tokenizer.bos_token = self.tokenizer.bos_token if self.tokenizer.bos_token else '[CLS]'
     82                 self.tokenizer.eos_token = self.tokenizer.eos_token if self.tokenizer.eos_token else '[SEP]'

KeyError: 'Fail to load the model from /data/hitz02/checkpoints/lcfs_atepc_cdw_apcacc_89.57_apcf1_72.5_atef1_78.53/! the checkpoint is broken, or maybe the checkpoint is not compatible with this version.'
yangheng95 commented 3 years ago

Downloading 1jjaAQM6F9s_IEXNpaY-bQF9EOrhq0PBD into ./checkpoints.json... Done. Available ATEPC model checkpoints for Version:1.1.7a1 (this version)

  1. Checkpoint Name: chinese Model: LCF-ATEPC Dataset:
    Version: 1.1 Description: Author: H, Yang (yangheng@m.scnu.edu.cn)

  2. Checkpoint Name: Laptop14 Model: Dataset:
    Version: 1.0.7.1 Description:Trained on T4 Author: M, Xu (xumayi@m.scnu.edu.cn)

  3. Checkpoint Name: english Model: LCF-ATEPC Dataset:
    Version: 1.1 Description: Author: H, Yang (yangheng@m.scnu.edu.cn)

  4. Checkpoint Name: HELP-WANTED Model: Dataset:
    Version: 0.0.0 Description:You can help us by sharing checkpoints (e.g. models trained on you own datasets) with community. Author:

    Downloading checkpoint:english from Google Drive... Notice: The pretrained model are used for testing, neither trained using fine-tuned the hyper-parameters nor trained with enough steps, it is recommended to train the model on your own custom datasets Checkpoint already downloaded, skip... TShirt dataset is integrated dataset from: https://github.com/yangheng95/ABSADatasets Seems datasets downloaded /content/integrated_datasets Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight']

    • This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
    • This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Using bos_token, but it is not set yet. Using eos_token, but it is not set yet. 100%|██████████| 1870/1870 [00:04<00:00, 417.59it/s, convert examples to features] 100%|██████████| 470/470 [00:01<00:00, 414.42it/s, convert examples to features] 2021-08-24 14:16:06,336 INFO: >>> model: <class 'pyabsa.core.atepc.models.lcf_atepc.LCF_ATEPC'> --> Active 2021-08-24 14:16:06,340 INFO: >>> optimizer: adamw --> Active 2021-08-24 14:16:06,343 INFO: >>> learning_rate: 2e-05 --> Active 2021-08-24 14:16:06,346 INFO: >>> pretrained_bert: bert-base-uncased --> Active 2021-08-24 14:16:06,348 INFO: >>> use_bert_spc: False --> Active 2021-08-24 14:16:06,350 INFO: >>> max_seq_len: 80 --> Active 2021-08-24 14:16:06,353 INFO: >>> SRD: 3 --> Active 2021-08-24 14:16:06,356 INFO: >>> use_syntax_based_SRD: False --> Active 2021-08-24 14:16:06,358 INFO: >>> dropout: 0.5 --> Active 2021-08-24 14:16:06,362 INFO: >>> l2reg: 5e-05 --> Active 2021-08-24 14:16:06,364 INFO: >>> num_epoch: 3 --> Active 2021-08-24 14:16:06,366 INFO: >>> batch_size: 16 --> Active 2021-08-24 14:16:06,370 INFO: >>> seed: 1 --> Active 2021-08-24 14:16:06,372 INFO: >>> embed_dim: 768 --> Active 2021-08-24 14:16:06,374 INFO: >>> hidden_dim: 768 --> Active 2021-08-24 14:16:06,377 INFO: >>> polarities_dim: 3 --> Active 2021-08-24 14:16:06,379 INFO: >>> log_step: 20 --> Active 2021-08-24 14:16:06,383 INFO: >>> gradient_accumulation_steps: 1 --> Active 2021-08-24 14:16:06,385 INFO: >>> dynamic_truncate: True --> Active 2021-08-24 14:16:06,387 INFO: >>> evaluate_begin: 2 --> Active 2021-08-24 14:16:06,391 INFO: >>> dataset_item: ['TShirt'] --> Active 2021-08-24 14:16:06,393 INFO: >>> dataset_name: TShirt --> Active 2021-08-24 14:16:06,395 INFO: >>> dataset_file: {'train': ['/content/integrated_datasets/atepc_datasets/TShirt/Menstshirt_Train.xml.seg.atepc'], 'test': ['/content/integrated_datasets/atepc_datasets/TShirt/Menstshirt_Test_Gold.xml.seg.atepc']} --> Active 2021-08-24 14:16:06,398 INFO: >>> device: cuda:0 --> Active 2021-08-24 14:16:06,407 INFO: >>> device_name: Tesla K80 --> Active 2021-08-24 14:16:06,410 INFO: >>> model_name: lcf_atepc --> Active 2021-08-24 14:16:06,414 INFO: >>> Version: 1.1.7a1 --> Active 2021-08-24 14:16:06,417 INFO: >>> save_mode: 1 --> Active 2021-08-24 14:16:06,420 INFO: >>> model_path_to_save: /content/checkpoints --> Active 2021-08-24 14:16:06,423 INFO: >>> num_labels: 6 --> Active 2021-08-24 14:16:06,425 INFO: >>> lcf: cdw --> Default 2021-08-24 14:16:06,429 INFO: >>> window: lr --> Default 2021-08-24 14:16:06,430 INFO: >>> initializer: xavieruniform --> Default 2021-08-24 14:16:06,433 INFO: >>> srd_alignment: True --> Default 2021-08-24 14:16:06,583 INFO: Running training for Aspect Term Extraction 2021-08-24 14:16:06,607 INFO: Num examples = 1870 2021-08-24 14:16:06,609 INFO: Batch size = 16 2021-08-24 14:16:06,610 INFO: Num steps = 348 100%|██████████| 117/117 [03:00<00:00, 1.54s/it, Epoch:0 | No evaluation until epoch:2] 100%|██████████| 117/117 [02:59<00:00, 1.54s/it, Epoch:1 | No evaluation until epoch:2] 100%|██████████| 117/117 [06:10<00:00, 3.17s/it, Epoch:2 | loss_apc:0.2974 | loss_ate:0.0539 | APC_ACC: 88.94(max:88.94) | APC_F1: 60.13(max:60.13) | ATE_F1: 78.18(max:78.18)]2021-08-24 14:28:17,360 INFO: -------------------------------------Training Summary------------------------------------- 2021-08-24 14:28:17,363 INFO: Max APC Acc: 88.94000 Max APC F1: 60.13000 Max ATE F1: 78.18000 Accumulated Loss: 477.69427712261677 2021-08-24 14:28:17,366 INFO: -------------------------------------Training Summary------------------------------------- Training finished, we hope you can share your checkpoint with everybody, please see: https://github.com/yangheng95/PyABSA#how-to-share-checkpoints-eg-checkpoints-trained-on-your-custom-dataset-with-community 2021-08-24 14:28:17,370 INFO: >>> model: <class 'pyabsa.core.atepc.models.lcf_atepc.LCF_ATEPC'> --> Active 2021-08-24 14:28:17,380 INFO: >>> optimizer: adamw --> Active 2021-08-24 14:28:17,383 INFO: >>> learning_rate: 2e-05 --> Active 2021-08-24 14:28:17,385 INFO: >>> pretrained_bert: bert-base-uncased --> Active 2021-08-24 14:28:17,387 INFO: >>> use_bert_spc: False --> Active 2021-08-24 14:28:17,392 INFO: >>> max_seq_len: 80 --> Active 2021-08-24 14:28:17,395 INFO: >>> SRD: 3 --> Active 2021-08-24 14:28:17,397 INFO: >>> use_syntax_based_SRD: False --> Active 2021-08-24 14:28:17,399 INFO: >>> lcf: cdw --> Active 2021-08-24 14:28:17,403 INFO: >>> dropout: 0.5 --> Active 2021-08-24 14:28:17,406 INFO: >>> l2reg: 5e-05 --> Active 2021-08-24 14:28:17,409 INFO: >>> num_epoch: 3 --> Active 2021-08-24 14:28:17,411 INFO: >>> batch_size: 16 --> Active 2021-08-24 14:28:17,414 INFO: >>> seed: 1 --> Active 2021-08-24 14:28:17,417 INFO: >>> embed_dim: 768 --> Active 2021-08-24 14:28:17,419 INFO: >>> hidden_dim: 768 --> Active 2021-08-24 14:28:17,424 INFO: >>> polarities_dim: 3 --> Active 2021-08-24 14:28:17,426 INFO: >>> log_step: 20 --> Active 2021-08-24 14:28:17,431 INFO: >>> gradient_accumulation_steps: 1 --> Active 2021-08-24 14:28:17,437 INFO: >>> dynamic_truncate: True --> Active 2021-08-24 14:28:17,440 INFO: >>> evaluate_begin: 2 --> Active 2021-08-24 14:28:17,449 INFO: >>> dataset_item: ['TShirt'] --> Active 2021-08-24 14:28:17,450 INFO: >>> dataset_name: TShirt --> Active 2021-08-24 14:28:17,451 INFO: >>> dataset_file: {'train': ['/content/integrated_datasets/atepc_datasets/TShirt/Menstshirt_Train.xml.seg.atepc'], 'test': ['/content/integrated_datasets/atepc_datasets/TShirt/Menstshirt_Test_Gold.xml.seg.atepc']} --> Active 2021-08-24 14:28:17,453 INFO: >>> device: cuda:0 --> Active 2021-08-24 14:28:17,455 INFO: >>> device_name: Tesla K80 --> Active 2021-08-24 14:28:17,456 INFO: >>> model_name: lcf_atepc --> Active 2021-08-24 14:28:17,458 INFO: >>> Version: 1.1.7a1 --> Active 2021-08-24 14:28:17,460 INFO: >>> save_mode: 1 --> Active 2021-08-24 14:28:17,461 INFO: >>> model_path_to_save: /content/checkpoints --> Active 2021-08-24 14:28:17,463 INFO: >>> num_labels: 6 --> Active 2021-08-24 14:28:17,465 INFO: >>> max_test_metrics: {'max_apc_test_acc': 88.94, 'max_apc_test_f1': 60.13, 'max_ate_test_f1': 78.18} --> Active 2021-08-24 14:28:17,466 INFO: >>> metrics_of_this_checkpoint: {'apc_acc': 88.94, 'apc_f1': 60.13, 'ate_f1': 78.18} --> Active 2021-08-24 14:28:17,468 INFO: >>> window: lr --> Default 2021-08-24 14:28:17,470 INFO: >>> initializer: xavieruniform --> Default 2021-08-24 14:28:17,471 INFO: >>> srd_alignment: True --> Default

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight']

I test on my environ and colab, find no error, I notice you actual use the 1.1.1 version of pyabsa. I upload a test version 1.1.7a1, you can try it.

hitz02 commented 3 years ago

Got Value Error now in v1.1.9 Dont know how but the training logs still show version 1.1.7.

config = ATEPCConfigManager.get_atepc_config_english()
checkpoint_path = 'lcf_atepc_cdw_apcacc_82.77_apcf1_72.41_atef1_80.97'

config.model = ATEPCModelList.LCFS_ATEPC
config.evaluate_begin = 5
config.num_epoch = 6
config.log_step = 100
config.seed = 2
data_path = '/data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt'

aspect_extractor = ATEPCTrainer(config=config,
                           dataset=data_path,
                           from_checkpoint=checkpoint_path,
                           checkpoint_save_mode=1,
                           auto_device='cuda'
                           )

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Using bos_token, but it is not set yet.
Using eos_token, but it is not set yet.
100%|██████████| 1870/1870 [00:19<00:00, 95.56it/s, convert examples to features] 
100%|██████████| 470/470 [00:04<00:00, 95.33it/s, convert examples to features] 
2021-08-25 12:30:25,915 INFO: >>> model: <class 'pyabsa.core.atepc.models.lcfs_atepc.LCFS_ATEPC'>  --> Active
2021-08-25 12:30:25,916 INFO: >>> optimizer: adamw  --> Active
2021-08-25 12:30:25,917 INFO: >>> learning_rate: 2e-05  --> Active
2021-08-25 12:30:25,918 INFO: >>> pretrained_bert: bert-base-uncased  --> Active
2021-08-25 12:30:25,919 INFO: >>> use_bert_spc: False  --> Active
2021-08-25 12:30:25,920 INFO: >>> max_seq_len: 80  --> Active
2021-08-25 12:30:25,921 INFO: >>> SRD: 3  --> Active
2021-08-25 12:30:25,922 INFO: >>> dropout: 0.5  --> Active
2021-08-25 12:30:25,922 INFO: >>> l2reg: 5e-05  --> Active
2021-08-25 12:30:25,923 INFO: >>> num_epoch: 6  --> Active
2021-08-25 12:30:25,924 INFO: >>> batch_size: 16  --> Active
2021-08-25 12:30:25,925 INFO: >>> seed: 2  --> Active
2021-08-25 12:30:25,926 INFO: >>> embed_dim: 768  --> Active
2021-08-25 12:30:25,926 INFO: >>> hidden_dim: 768  --> Active
2021-08-25 12:30:25,927 INFO: >>> polarities_dim: 3  --> Active
2021-08-25 12:30:25,929 INFO: >>> log_step: 100  --> Active
2021-08-25 12:30:25,930 INFO: >>> gradient_accumulation_steps: 1  --> Active
2021-08-25 12:30:25,930 INFO: >>> dynamic_truncate: True  --> Active
2021-08-25 12:30:25,931 INFO: >>> srd_alignment: True  --> Active
2021-08-25 12:30:25,931 INFO: >>> evaluate_begin: 5  --> Active
2021-08-25 12:30:25,932 INFO: >>> dataset_item: ['/data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt']  --> Active
2021-08-25 12:30:25,932 INFO: >>> dataset_name: custom_dataset  --> Active
2021-08-25 12:30:25,933 INFO: >>> dataset_file: {'train': ['/data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt/Menstshirt_Train.xml.seg.atepc'], 'test': ['/data/hitz02/ABSADatasets/datasets/atepc_datasets/TShirt/Menstshirt_Test_Gold.xml.seg.atepc']}  --> Active
2021-08-25 12:30:25,933 INFO: >>> device: cuda  --> Active
2021-08-25 12:30:25,934 INFO: >>> device_name: NVIDIA GeForce GTX 1080 Ti  --> Active
2021-08-25 12:30:25,935 INFO: >>> model_name: lcfs_atepc  --> Active
2021-08-25 12:30:25,935 INFO: >>> Version: 1.1.7  --> Active
2021-08-25 12:30:25,936 INFO: >>> save_mode: 1  --> Active
2021-08-25 12:30:25,936 INFO: >>> model_path_to_save: /data/hitz02/checkpoints  --> Active
2021-08-25 12:30:25,937 INFO: >>> num_labels: 6  --> Active
2021-08-25 12:30:25,937 INFO: >>> use_syntax_based_SRD: False  --> Default
2021-08-25 12:30:25,938 INFO: >>> lcf: cdw  --> Default
2021-08-25 12:30:25,938 INFO: >>> window: lr  --> Default
2021-08-25 12:30:25,939 INFO: >>> initializer: xavier_uniform_  --> Default
Checkpoint loaded!
2021-08-25 12:30:26,486 INFO: ***** Running training for Aspect Term Extraction *****
2021-08-25 12:30:26,487 INFO:   Num examples = 1870
2021-08-25 12:30:26,488 INFO:   Batch size = 16
2021-08-25 12:30:26,488 INFO:   Num steps = 696
100%|██████████| 117/117 [01:36<00:00,  1.21it/s, Epoch:0 | No evaluation until epoch:5]
100%|██████████| 117/117 [01:36<00:00,  1.21it/s, Epoch:1 | No evaluation until epoch:5]
100%|██████████| 117/117 [01:35<00:00,  1.22it/s, Epoch:2 | No evaluation until epoch:5]
100%|██████████| 117/117 [01:36<00:00,  1.21it/s, Epoch:3 | No evaluation until epoch:5]
100%|██████████| 117/117 [01:35<00:00,  1.22it/s, Epoch:4 | No evaluation until epoch:5]
 12%|█▏        | 14/117 [00:26<03:14,  1.88s/it]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-13-d4b7ed1bdcf7> in <module>
      3                            from_checkpoint=checkpoint_path,
      4                            checkpoint_save_mode=1,
----> 5                            auto_device='cuda'
      6                            )

/data/hitz02/env/lib/python3.6/site-packages/pyabsa/functional/trainer/trainer.py in __init__(self, config, dataset, from_checkpoint, checkpoint_save_mode, auto_device)
    101             config.model_path_to_save = None
    102 
--> 103         self.train()
    104 
    105     def train(self):

/data/hitz02/env/lib/python3.6/site-packages/pyabsa/functional/trainer/trainer.py in train(self)
    113             config.seed = s
    114             if self.checkpoint_save_mode:
--> 115                 model_path.append(self.train_func(config, self.from_checkpoint, self.logger))
    116             else:
    117                 # always return the last trained model if dont save trained model

/data/hitz02/env/lib/python3.6/site-packages/pyabsa/core/atepc/training/atepc_trainer.py in train4atepc(opt, from_checkpoint_path, logger)
    353     load_checkpoint(trainer, from_checkpoint_path)
    354 
--> 355     return trainer.run()

/data/hitz02/env/lib/python3.6/site-packages/pyabsa/core/atepc/training/atepc_trainer.py in run(self)
    207                                 )
    208 
--> 209                                 save_model(self.opt, self.model, self.tokenizer, save_path)
    210 
    211                         current_apc_test_acc = apc_result['apc_test_acc']

/data/hitz02/env/lib/python3.6/site-packages/pyabsa/utils/file_utils.py in save_model(opt, model, tokenizer, save_path)
    237             # torch.save(self.model.cpu().state_dict(), save_path + self.opt.model_name + '.state_dict')  # save the state dict
    238         torch.save(model.state_dict(), save_path + opt.model_name + '.state_dict')  # save the state dict
--> 239         pickle.dump(opt, open(save_path + opt.model_name + '.config', mode='wb'))
    240         pickle.dump(tokenizer, open(save_path + opt.model_name + '.tokenizer', mode='wb'))
    241         save_args(opt, save_path + opt.model_name + '.args.txt')

ValueError: binary mode doesn't take an encoding argument