Script for LayoutXLM pretraining

[Armandgrd]

Hello, I would like to continue the pre-training of the LayoutXLM on my own domain data, nevertheless I only found scripts to perform fine-tuning on particular tasks (funsd and xfun). Is it possible for you to provide the script which was used for the pre-training step , it would be incredibly useful ? I am also interested in how you perform the visual tasks described in the paper (MMLM, TLM, XlCo) .

By the way, thanks for your great work on this models.

[wolfshow]

@Armandgrd We have no bandwidth for the pre-training code release right now, but have plan for this in the future.

[Armandgrd]

Hello, thanks for your answer @wolfshow ,

I rewrote the pretraining code anyway, It would be helpful even if you could provide the code as it is now, even with a non-working code , it's just to check if I implemented something wrong.

Is it possible to get the parameters of the pre-trained model, for now, I am using the fine-tuned version and the parameters of the MLM head are unfortunately initialized randomly, which is a shame imo.

I would also be pleased to contribute 🙂 , if you need help on something and that can help you free some time.

[valentinkoe]

@Armandgrd would you be willing to share your version of the pre-training code?

[Armandgrd]

@valentindey sure, have a look at this gist let me know if you need anything else. Hope it helps

[valentinkoe]

@Armandgrd thanks a lot for sharing! It definitely helps. Do you happen to also have the code for preparing data for the different training tasks?

[Armandgrd]

@valentindey Un fortunately, I only implemented the part for the Multilingual Masked Visual-Language Modeling, and the code for preparing data is highly dependant on the format of your documents, I am not sure it will help.

The high level is:

1. read document images
2. detect word and boxes using pdfplumber (the orginal authors used PyMuPDF) including CLF in a CustomDocDataset and data loader
3. tokenize text, create masks
4. fit and evaluate cross entropy

It would be great to have the author code for the Text-Image Alignment and Matching since it is the main innovation of their paper. I plan to implement them in the near future:

• for alignment: you can adapt token classification and use just two labels, [masked, not masked] and apply a masking function on the images.
• for matching, I'm not sure, maybe a sequence classification with a binary label (same source document or not), while shuffling some images between documents. it would be great if @wolfshow could explain more precisely what was done here.