search

NielsRogge / Transformers-Tutorials

This repository contains demos I made with the Transformers library by HuggingFace.
MIT License
9.15k stars 1.42k forks source link

How to get key value pairs of words and its predicted label in layoutlm model instead of showing it in an image? #82

Open vishal-nayak1 opened 2 years ago

vishal-nayak1 commented 2 years ago

Model I am using (LayoutLM): Is there any way to get key value pairs from BIOES tagged prediction on single image?

sankalp-wns commented 2 years ago

@vishal-nayak1 Have you figured it out?

vishal-nayak1 commented 2 years ago

@sankalp-wns Yes, I wrote a inference script for it to extract key value pairs.I cannot share the code due to company policy but will share some ideas how you can implemented it.

  1. convert BIOES to BIO tag (if bioes scheme used)- https://github.com/jiesutd/NCRFpp/blob/master/utils/tagSchemeConverter.py
  2. convert BIO tag to original text- https://medium.com/analytics-vidhya/bio-tagged-text-to-original-text-99b05da6664 I hope it might be helpful :-)
jyotiyadav94 commented 2 years ago

Hi @vishal-nayak1 , Thank you very much for sharing your ideas. I am working on the use case where I have to get the questions as keys and answers as values as described in the below image .

Image 1 image

I have tried the ideas which you have given in the above comments. I was able to get the keys and values in this way Image 2 image

However do you have any ideas to print the predicted out as described in the Image 1 where all the questions are described in the columns and their respective values

jyotiyadav94 commented 2 years ago

@vishal-nayak1 how do you get the BIOES tagged predictions from layoutLMV3 inference? I am using the below code - https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LayoutLMv3

vishal-nayak1 commented 2 years ago

Hi @jyotiyadav94, With reference to your image 1, I would suggest you can define all your questions as NER labels(as key) and its answers as text to be extracted. In your current implementation you need to use entity linking to map questions to their corresponding answers.

vishal-nayak1 commented 2 years ago

BIOES, BIO etc. are different scheme to represent your ner labels for multiwords.You can use any of these. You can refer to this link for more information- https://github.com/jiesutd/NCRFpp/blob/master/utils/tagSchemeConverter.py

jyotiyadav94 commented 2 years ago

@vishal-nayak1 , Thank you.