Slow for long PDFs

[loganlebanoff]

Hi, I have a PDF that is 700 pages long. It takes 30 seconds to extract paragraphs from it when I used the docker-compose setup.

Is there a way to speed this up? Perhaps by using GPU or parallelizing computation among multiple workers?

[gabriel-piles]

Hello,

Thank you for your interest.

Unfortunately, it is not easy to speed up the process. Let me explain.

For 700 pages the time for each step is as follow:

• 3 seconds: Converting the PDF to an XML using Poppler
• 10 seconds: Convert the XML to a Python class and extract the features
• 9 seconds: Using LightGBM to segment the PDF. LightGBM, even inside the docker container, is using all the cores in the machine so it is not possible to speed up with parallelizing, and in our tests, it is not improving the time using GPU.
• 6 seconds: Get the results back

So, theoretically, the second and fourth steps can be speed up but it requires some work that we are going to do in the future. Unfortunately, we can not give right now an estimation of when this is going to happen.

If you are interested in contributing to the project, you are welcome.

Thank you

[loganlebanoff]

I see. Thank you for the detailed explanation!