search

jpWang / LiLT

Official PyTorch implementation of LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding (ACL 2022)
MIT License
335 stars 40 forks source link

fail to reproduce the result of language-specific (for example, ZH) relation extraction on XFUND #13

Closed zisang0210 closed 2 years ago

zisang0210 commented 2 years ago

When conducting the experiment of language-specific fine-tuning on XFUND, the obtained f1 score is only 0.6179 using the following command, which has a huge gap compared with the reported f1 score of 0.7297. The XFUND dataset and pretrained lilt-infoxlm-base are downloaded from the urls mentioned in the README.md. Are there any additional steps to reproduce the experiment?

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 examples/run_xfun_re.py \ --model_name_or_path lilt-infoxlm-base \ --tokenizer_name xlm-roberta-base \ --output_dir ls_re_xfund_zh_lilt-infoxlm-base \ --do_train \ --do_eval \ --lang zh \ --max_steps 5000 \ --per_device_train_batch_size 8 \ --learning_rate 6.25e-6 \ --warmup_ratio 0.1 \ --fp16

jpWang commented 2 years ago

Hi, have you cloned the latest repo and followed the provided instructions strictly? I have re-checked it and the reproduced performance is expected.

zisang0210 commented 2 years ago

The problem is solved by reconfiguring the running environment, thanks a lot for your reply. By the way, pytorch should be installed via pip instead of conda, otherwise an error will be reported when running: ImportError: libc10_cuda.so: cannot open shared object file: No such file or directory