Open jacobmarks opened 1 year ago
https://github.com/facebookresearch/nougat/issues/11#issue-1874120428
I meet the same problem. There is nothing but
[MISSING_PAGE_EMPTY:1]
[MISSING_PAGE_EMPTY:2]
[MISSING_PAGE_EMPTY:3]
in the mmd file.
Hi @jacobmarks After some investigation I found two reasons:
I will look into the failure detection heuristic again. Thank you for brining this to my attention.
Thanks for looking into this @lukas-blecher ! Really appreciate your prompt response
I am also getting a lot of 'MISSING_PAGE_FAIL:{n}]' on output file and 'WARNING:root:Skipping page {n} due to repetitions.' in terminal.
I'm experiencing a similar issue of missing pages during PDF to Markdown conversion, but with some nuances.
nougat \path\to\file.pdf -o \path\to\folder\ --markdown
Let me know if you need any more info.
I'm experiencing a similar issue of missing pages during PDF to Markdown conversion, but with some nuances.
Issue Details
- Paper Tested (original, complete pdf): Springer Article
- Command Used:
nougat \path\to\file.pdf -o \path\to\folder\ --markdown
- Expected Behavior: Complete conversion of all pages.
- Actual Behavior: Missed pages: 6, 8-10, 21, 23, 26-33, 35-36, 39, 46-49.
- I then created a new PDF paper with only the missing pages and tested that. None of the new pdf pages were processed (it missed all of them)
- Output (for the pdf with only missing pages): C:\Users\Me>nougat C:\Users\Me\Downloads\papers\missed.pdf -o C:\Users\Me\Downloads\papers --markdown C:\Users\Me\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ..\aten\src\ATen\native\TensorShape.cpp:3484.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] 0%| | 0/21 [00:00<?, ?it/s]WARNING:root:Found repetitions in sample 0 INFO:root:Processing file C:\Users\Me\Downloads\papers\missed.pdf with 21 pages 5%|███▉ | 1/21 [00:08<02:54, 8.70s/it]WARNING:root:Found repetitions in sample 0 WARNING:root:Skipping page 2 due to repetitions. 10%|███████▉ | 2/21 [00:28<04:45, 15.05s/it]WARNING:root:Found repetitions in sample 0 14%|███████████▊ | 3/21 [00:35<03:24, 11.36s/it]WARNING:root:Found repetitions in sample 0 19%|███████████████▊ | 4/21 [00:41<02:36, 9.20s/it]W...
Additional Notes
- This issue is consistently reproducible.
- Running the same paper through the online HuggingFace demo resulted in fewer missing pages.
Let me know if you need any more info.
I actually noticed this as well, that the quality of the output generated by Nougat was better in the Huggingface demo than on my own computer. Maybe this is a false positive though?
Remember that Nougat was trained on Arxiv scientific papers in the STEM field, so if you feed it a magazine article in PDF format that only includes text and some graphics, don't be surprised if it fails here and there. Out-of-domain content seems to not be super robust yet. If your doc includes content that is not similar (enough) to the training data it will not work correctly, just like a dog you've taught to sit will not necessarily stay.
I actually noticed this as well, that the quality of the output generated by Nougat was better in the Huggingface demo than on my own computer. Maybe this is a false positive though?
Remember that Nougat was trained on Arxiv scientific papers in the STEM field, so if you feed it a magazine article in PDF format that only includes text and some graphics, don't be surprised if it fails here and there. Out-of-domain content seems to not be super robust yet. If your doc includes content that is not similar (enough) to the training data it will not work correctly, just like a dog you've taught to sit will not necessarily stay.
Did you see my doc? It's pretty academic and Arxiv-like. I wonder if it's the architecture of my old hardware that's the issue.
I actually noticed this as well, that the quality of the output generated by Nougat was better in the Huggingface demo than on my own computer. Maybe this is a false positive though? Remember that Nougat was trained on Arxiv scientific papers in the STEM field, so if you feed it a magazine article in PDF format that only includes text and some graphics, don't be surprised if it fails here and there. Out-of-domain content seems to not be super robust yet. If your doc includes content that is not similar (enough) to the training data it will not work correctly, just like a dog you've taught to sit will not necessarily stay.
Did you see my doc? It's pretty academic and Arxiv-like. I wonder if it's the architecture of my old hardware that's the issue.
I did see your doc. Looks like it should be processed perfectly fine? Did the shape of the LaTeX rendered integral sign change drastically at one point? Unlikely to be a hardware issue IMO, the processes being run are the same, so the only difference with better hardware (video card) would be more VRAM and therefore lower inferencing time, not necessarily changing quality of output.
Have you tried with the Huggingface demo? How did the output of the Huggingface demo compare with the results you achieved?
I'm experiencing a similar issue of missing pages during PDF to Markdown conversion, but with some nuances.
System Information
- Nougat Version: 0.1.4
- OS: Windows 11
- GPU: NVIDIA 1050 mobile with 4GB VRAM
- RAM: 16GB
- Pytorch Version: 2.0.1+cu118
- NVIDIA-SMI: Driver version 537.13
- CUDA Version: 12.2
Issue Details
- Paper Tested (original, complete pdf): Springer Article
- Command Used:
nougat \path\to\file.pdf -o \path\to\folder\ --markdown
- Expected Behavior: Complete conversion of all pages.
- Actual Behavior: Missed pages: 6, 8-10, 21, 23, 26-33, 35-36, 39, 46-49.
- I then created a new PDF paper with only the missing pages and tested that. None of the new pdf pages were processed (it missed all of them)
- Output (for the pdf with only missing pages): C:\Users\Me>nougat C:\Users\Me\Downloads\papers\missed.pdf -o C:\Users\Me\Downloads\papers --markdown C:\Users\Me\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ..\aten\src\ATen\native\TensorShape.cpp:3484.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] 0%| | 0/21 [00:00<?, ?it/s]WARNING:root:Found repetitions in sample 0 INFO:root:Processing file C:\Users\Me\Downloads\papers\missed.pdf with 21 pages 5%|███▉ | 1/21 [00:08<02:54, 8.70s/it]WARNING:root:Found repetitions in sample 0 WARNING:root:Skipping page 2 due to repetitions. 10%|███████▉ | 2/21 [00:28<04:45, 15.05s/it]WARNING:root:Found repetitions in sample 0 14%|███████████▊ | 3/21 [00:35<03:24, 11.36s/it]WARNING:root:Found repetitions in sample 0 19%|███████████████▊ | 4/21 [00:41<02:36, 9.20s/it]W...
Additional Notes
- This issue is consistently reproducible.
- Running the same paper through the online HuggingFace demo resulted in fewer missing pages.
Let me know if you need any more info.
Hi, thanks for the detailed report. However I was unable to reproduce the it. My best guess is that it's the GPU. Can you try to convert some of the failed pages with CPU only (set batch size to 0 -b 0
)
I have noticed some very slight differences when running on full precision but not so drastic as you are describing them
Edit: Was able to reproduce on CPU. Will investigate now, thanks!
I was able to confirm that this is again a case of a false positive failure detection. I've added the --no-skipping
flag. Please try and rerun your conversion with this flag on, thank you
@lukas-blecher what specifically does the --no-skipping
flag do (other than not skip pages)? I'm asking since I'm also interested, since this sort of issue happens to me quite a bit too^^ even though your reply was to @sm18lr88
In short, it won't apply the failure detection heuristic described in the paper. I still haven't fully grasped the problem at hand but for some reason pytorch gives different values depending on the device you're using to compute. Since I chose the the threshold etc on the specific GPU type I used to test the model, I never noticed this issue myself.
But what this also means is that true positives won't be caught as well. So you might get a lot of repetitions for out of domain PDFs (plus the computation time will be longer because we aren't stopping in the middle of the generation anymore)
In short, it won't apply the failure detection heuristic described in the paper. I still haven't fully grasped the problem at hand but for some reason pytorch gives different values depending on the device you're using to compute. Since I chose the the threshold etc on the specific GPU type I used to test the model, I never noticed this issue myself.
But what this also means is that true positives won't be caught as well. So you might get a lot of repetitions for out of domain PDFs (plus the computation time will be longer because we aren't stopping in the middle of the generation anymore)
Is this also why it seems that the huggingface demo seems to give me the best results? Where does that run? I'm pretty convinced that this does not run locally.
So hardware seems to have an effect on the quality of the output, correct?
Also, repetitions for out of domain documents, this is already a known issue so with this new flag, will this get worse, stay the same or improve?
The hardware doesn't really change the output but it does change the repetition / failure detection. So you should get the same results as the HF demo.
What will change for out of domain documents is that the repetition will not be detected during the generation. There are still some rule based postprocessing functions that will detect some of them though.
As I still continue to experience missing pages (which I'm sure will improve over time), I find myself turning to Claude (v2) and asking it to convert pdfs into the format I desire. It's very good at it.
FYI, I believe Claude 2 preprocesses PDF files using Mathpix (at least on the official website).
FYI, I believe Claude 2 preprocesses PDF files using Mathpix (at least on the official website).
Well, for whatever reason it was doing an excellent job. IDK if it was my GPU/CUDA compatibility or what.
I have the same problem.
same issue using nougat_api, can I use --no-skipping somehow?
how to change tmp o setting with the api?
Tried applying to individual pages of the PDF for EPR paper https://cds.cern.ch/record/405662/files/PhysRev.47.777.pdf.
While the first page works and I get a print out of the text, pages 2-4 don't work. I get errors like this: