search

microsoft / unilm

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
https://aka.ms/GeneralAI
MIT License
19.62k stars 2.5k forks source link

LayoutLM using roberta as initialization #555

Open jianglong-he-Infrrd opened 2 years ago

jianglong-he-Infrrd commented 2 years ago

Hi, I think the released LayoutLM v1 model in huggingface is initialized with BERT weights. I do see in the paper it states that initializing with roberta weights gives better results. IS there a reason for not releasing LayouLM v1 model with roberta initialization?

wolfshow commented 2 years ago

https://huggingface.co/microsoft/layoutlm-base-cased

jianglong-he-Infrrd commented 2 years ago

Thanks for the clarification! Is there also an large model LayoutLM based on roberta initialization?