현재 까지 우리팀 Model 정리

[sangmandu]

구성원	model	batch size	특징	Eval	LB
손희락	klue/roberta-large	128	train 시 concat 데이터 사용	65.00	61.670
손희락	klue/roberta-large	128	train 시 k-fold & concat 데이터 사용	68.653, 66.031, 66.11, 66.229, 64.678	66.250
심우창	klue/roberta-large	256	MRC mask + concat 데이터 사용	70.00	63.750

roboerta-large batch 128 train 시 concat 데이터 사용

python train.py 
--do_train 
--project_name mrc_concat_data_train 
--model_name_or_path klue/roberta-large 
--run_name roberta-large_batch_128_concat_5 
--with_inference False 
--dataset_name concat 
--per_device_train_batch_size 16 
--gradient_accumulation_steps 8 
--num_train_epochs 20 

python inference.py 
--do_predict 
--project_name mrc_concat_data_train 
--finetuned_mrc_model_path ../output/mrc_concat_data_train/roberta-large_batch_128_concat_5 
--run_name roberta-large_batch_128_concat_5 
--elastic_index_name preprocess-wiki-index

roboerta-large batch 128 train 시 k-fold & concat 데이터 사용

python train.py 
--do_train 
--project_name mrc_concat_data_train 
--model_name_or_path klue/roberta-large 
--run_name roberta-large_batch_128_fold
--with_inference False 
--dataset_name concat 
--per_device_train_batch_size 16 
--gradient_accumulation_steps 8 
--num_train_epochs 20 
--fold True

for i in rnage(1, 6):
  python inference.py 
  --do_predict 
  --project_name mrc_concat_data_train 
  --finetuned_mrc_model_path ../output/mrc_concat_data_train/roberta-large_batch_128_fold/{i}
  --run_name roberta-large_batch_128_fold 
  --elastic_index_name preprocess-wiki-index

MRC mask roberta-large batch 256 & concat 데이터 사용

python train_mrc.py 
--project_name="question_mask" 
--run_name="robert-large-concat-gradient16-fold" 
--dataset_name="concat" 
--per_device_retrieval_train_batch_size=128  
--do_train=True 
--model_name_or_path='klue/roberta-large' 
--gradient_accumulation_steps=16

python inference.py
--do_predict=True 
--project_name="robert-large_concat_gradient16-result" 
--finetuned_mrc_model_path="/opt/ml/mrc-level2-nlp-04/output/question_mask/robert-large-concat-gradient16" 
--elastic_index_name="preprocess-wiki-index" 
--run_name="inference"