[Request] Implement CPU mode even when GPU system can support.

[RandomInternetPreson]

Hello, great model and thank you for all of your hard work!

I use the model in GPU mode on windows and it runs very well and I have implemented your model capabilities in an extension for Oobabooga's text-generation-webui: https://github.com/oobabooga/text-generation-webui

My extension: https://github.com/RandomInternetPreson/LucidWebSearch

There are a lot of people that use the text-generation-webui, of whom are vram limited. Is it possible (and within your scope of work and worth a look) to add a --cpu flag or something like that to force the model to run in the CPU only?

I have edited your code here: https://github.com/RandomInternetPreson/LucidWebSearch/blob/main/device_LowVramCPUMODE.py

This reduces the vram utilization by a significant amount, down to a little less than 5GB, but for many that 5GB is probably still a big chunk of their total vram. I'm not sure what else to edit, I've tried other edits, with none affecting the resulting vram utilization.

Even if it's not worth the hassle of putting in a --cpu flag, could you point me to which file I would need to edit to make it happen?

[lukas-blecher]

Yes, that's a good idea. Right now you can hack this by setting --batchsize 0

[RandomInternetPreson]

I have found that the margin spacing is one of the most influential variables to getting a good conversion. And if the quality of the text is too good it seems to introduce more errors. I would try to make the margins 1.25 inches and reduce the quality of the PDF. if it's a large sized file for a few pages, then the quality might be too high.

[RandomInternetPreson]

@lukas-blecher thank you! I'll give your suggestion a try, very much appreciate the feedback 😊

[xsank]

Is there anybody works well on the Mac M1 mps mode? The nougat model predict blocked on my computer...

[perryzjc]

Is there anybody works well on the Mac M1 mps mode? The nougat model predict blocked on my computer...

Same question