[Document Understanding] Can we support a new task on document understanding?

[jlia0]

Document Understanding

Some example models:

1. DiT: https://huggingface.co/microsoft/dit-large
2. LayoutLMv3: https://huggingface.co/microsoft/layoutlmv3-large
3. Donut: https://huggingface.co/docs/transformers/model_doc/donut

Reason for request

Document understanding is a very popular task which I couldn't find any supports for the web environment.

Some tasks include:

1. Key Information Extraction (KIE)
2. Document Layout Analysis (DLA)
3. Document Question Answering (DQA)
4. Optical Character Recognition (OCR)

[xenova]

Those do sound like quite interesting use-cases! Do you mind sharing example code for how you would use the models, as well as the inputs and expected outputs?

[jlia0]

Here's an example code using detectron2 and DiT on document layout analysis.

DiT Doc: https://huggingface.co/docs/transformers/v4.31.0/en/model_doc/dit HF Space: https://huggingface.co/spaces/imjliao/dit-document-layout-analysis/blob/main/app.py

[xenova]

The repo you shared is private, but I assume I can use this one: https://huggingface.co/spaces/nielsr/dit-document-layout-analysis

[jlia0]

The repo you shared is private, but I assume I can use this one: https://huggingface.co/spaces/nielsr/dit-document-layout-analysis

Oh yes sorry! I forgot it's my private repo. But you're correct, I am using that one as well.

How do you think we can include this to transformer.js? Seems like there is a dependency issue of detectron2...

[xenova]

Hmm, that might complicate things somewhat... Perhaps there is a JS library out there which is a suitable substitute?

[jlia0]

Hmm, that might complicate things somewhat... Perhaps there is a JS library out there which is a suitable substitute?

I don't see a JS library out there could do similar stuffs. But I found something that's worth checking out:

https://github.com/Unstructured-IO/unstructured-inference/blob/main/unstructured_inference/models/detectron2onnx.py

^^^ This is a working example of detectron2 using ONNXRuntime...

[xenova]

Just an update on this:

• https://github.com/xenova/transformers.js/pull/320 added support for Donut models (document parsing and document question answering)
• https://github.com/xenova/transformers.js/pull/375 will add support for TrOCR models (optical character recognition)

The other tasks (Key Information Extraction and Document Layout Analysis) might be slightly more difficult to add (due the their additional dependencies)... but we'll get there eventually :)