Accuracy is around base rate?

[dfayzur]

Describe Model I am using LayoutLM:

I adapted the run_classification.py file in the example of LayoutLM to my dataset for binary classification. I used pretrained LayoutLM from this repo. But almost all the time I get validation accuracy is arround base rate (60% in the case of the dataset), but one of two time I get accuracy around 80%. Here I want to mention, I set the same seed for simulation.

Then, I ran the run_classification.py for rvl-cdip data with only data of two class. The LayoutLM was pretrained on rvl-cdip dataset. So I was expecting to get high accuracy on validation dataset. But still the accuracy is around 50%. Although I calculated AUC, which is surprisingly 1.

My question: is the behavior normal? Can you please guide me to make the LayoutLM model working?

Thanks for your time.

[wolfshow]

@dfayzur did you fine-tune the model with your own dataset?

[dfayzur]

@wolfshow , yes I fine-tune the model with my own dataset (binary problem), and got accuracy around 60% (which is the base rate of my dataset). That means the fine-tune is not learning.

Then, I downloaded rvl-cdip data and took only two class, so that it matches to binary problem. Again the accuracy is around base rate 50% (because the two class I took are balanced.)

[wolfshow]

@dfayzur Can you show more details and the command you run for the fine-tuning?

[dfayzur]

@wolfshow the command I used for running the fine-tuning bellow:

python run_classification.py --model_name_or_path "/code/public_datasets/layoutlm/LayoutLM-Base-Uncased" \ --model_config "/code/public_datasets/layoutlm/LayoutLM-Base-Uncased/config.json" \ --data_dir '/code/public_datasets/rvl-cdip/layoutLM/' \ --model_type 'layoutlm' \ --output_dir '/code/public_datasets/output/layoutlm_cdip_2class/' \ --config_name "" \ --tokenizer_name "" \ --cache_dir "" \ --max_seq_length 512 \ --do_train True \ --do_eval True \ --do_test True \ --evaluate_during_training True \ --do_lower_case True \ --per_gpu_train_batch_size True \ --per_gpu_eval_batch_size True \ --gradient_accumulation_steps 1 \ --learning_rate 5e-5 \ --weight_decay 0.0 \ --adam_epsilon 1e-8 \ --max_grad_norm 1.0 \ --num_train_epochs 50.0 \ --max_steps -1 \ --warmup_steps 0 \ --logging_steps 50 \ --save_steps 50 \ --eval_all_checkpoints True \ --no_cuda False \ --overwrite_output_dir True \ --overwrite_cache False \ --seed 42 \ --tpu False \ --fp16 False \ --fp16_opt_level "O1" \ --local_rank =-1 \ --server_ip "" \ --server_port ""