How to use nougat with multiple GPUs or quantization?

[brando90]

How does one use nougat with multiple GPUs? I keep getting memory issues:

/lfs/skampere1/0/brando9/miniconda/envs/maf/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
  0%|                                                                                                                            | 0/11 [00:19<?, ?it/s]
Traceback (most recent call last):
  File "/lfs/skampere1/0/brando9/miniconda/envs/maf/bin/nougat", line 8, in <module>
    sys.exit(main())
  File "/lfs/skampere1/0/brando9/miniconda/envs/maf/lib/python3.10/site-packages/predict.py", line 167, in main
    model_output = model.inference(
  File "/lfs/skampere1/0/brando9/miniconda/envs/maf/lib/python3.10/site-packages/nougat/model.py", line 614, in inference
    logits = torch.stack(decoder_output.scores, 1).cpu().max(-1)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.93 GiB (GPU 0; 79.15 GiB total capacity; 30.13 GiB already allocated; 2.10 GiB free; 30.65 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
-> Cannot close object, library is destroyed. This may cause a memory leak!
OCRing with small model

[brando90]

or with quantization.

[brando90]

possible solution is to change batch size in nougat cmd e.g.,

        nougat $path_2_pdf --out $path2mmd --recompute --no-skipping --markdown --model 0.1.0-small --batchsize 8

[brando90]

https://github.com/facebookresearch/nougat/issues/147