search

jpWang / LiLT

Official PyTorch implementation of LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding (ACL 2022)
MIT License
335 stars 40 forks source link

Layout Analysis #11

Closed alejandrojcastaneira closed 1 year ago

alejandrojcastaneira commented 2 years ago

First of all thanks, for the great work! My question, could these models be adapted to the task of Layout Analysis, so that we could use them in datasets like Publaynet? In this sense, the models would need to output the probabilities of each pixel belonging to a given class, instead of the possible tags for each token.

jpWang commented 2 years ago

Hi, since LiLT does not introduce image information yet, it cannot directly output the probabilities of each pixel. However, maybe you can first use the OCR engine to get the OCR result and then classify each token into the categories of pixels contained in its corresponding box. In this way, it can assist the traditional visual model in dealing with layout analysis tasks.

mllife commented 6 months ago

Anyone has any update on this?