How to process a pdf to convert to mmd inside of python instead of bash? (instead of nougat path/to/file.pdf -o output_directory -m 0.1.0-base)

[brando90]

How to process a pdf to convert to mmd inside of python instead of bash? (instead of nougat path/to/file.pdf -o output_directory -m 0.1.0-base)

I see that the recommended command is

$ nougat path/to/file.pdf -o output_directory -m 0.1.0-base

but I want to process the pdfs in concurrently with threads (io bounded by nougat's slowness + my small batch size needed or it crashes due to OMM) because the following:

for path_2_pdf in "${original_path_2_pdfs[@]}"; do
    nougat $path2pdf -o output_directory -m 0.1.0-base
done

is too slow. I sometimes have 8 gpus with 80GBs available.

So how do I do the equivalent command where I pass a single path to the pdf and process it to convert the pdf to .mmd? (pdf -> mmd).

[qaiwiz]

You can use the docker. However, for that remember to use the fix here: https://github.com/facebookresearch/nougat/pull/208