search

microsoft / unilm

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
https://aka.ms/GeneralAI
MIT License
20.08k stars 2.55k forks source link

LayoutLM model only able to classify individual words instead of entire sections #1028

Open ghost opened 1 year ago

ghost commented 1 year ago

Model I am using (LayoutLM ...):

Here, I would like to develop a custom resume parser model that can accurately predict the sections for EDUCATION, SKILLS, and EXPERIENCE based on the resume. I have fine-tuned the LayoutLMv3 model on a custom dataset that is similar to the FUNSD dataset.

Although the LayoutLM model can predict education keywords, it only does so at the word level. For instance, if the resume states "My education is in computer engineering from LD College Ahmedabad," the model will label "computer" and "engineering" as EDUCATION. However, I aim to have all classified words in a single section rather than in individual word sections.

Therefore, here are some random screenshots of the LayoutLM model output. Screenshot from 2023-03-13 18-36-47

And here, I would like the output to include box coordinates for the EDUCATION section as well as the SKILLS section, identified by their respective keywords. Screenshot from 2023-03-13 18-32-05

Note: I have attempted to use the Layout Parser model with the PublayNet dataset. However, this model was unable to accurately predict and classify the sections for EDUCATION, SKILLS, EXPERIENCE, etc.

If there are any other models that would be suitable for my use case, please kindly suggest them. Thank you all for your help.

mllife commented 1 year ago

I also tried the code from hugging face but the output coming in term of per word labels, rather than block or section boxes. Please help.

Found this similar issue link - https://github.com/microsoft/unilm/issues/906

But I don't know how to use it in code. Can someone make example code that takes a image as input and give section box and labels.