search

Juqowel / GPU_For_T5

11 stars 1 forks source link

It doesn't seem to work for CPU mode #4

Open HerMajestyDrMona opened 4 days ago

HerMajestyDrMona commented 4 days ago

When I'm using "Force/Set CLIP Device" on ComfyUI, I can see the high 100% CPU usage (on all 16 cores) for a few seconds. With this extension, even though I'm loading "t5xxl_fp16.safetensors", the CPU usage never jumps high, so I assume it's still using my GPU and VRAM.

The extension is correctly enabled, because I can see messages:

T5 on Other Device is disabled for txt2img
T5 on Other Device is enabled for txt2img

What could be wrong? Maybe I don't understand the purpose of this extension? I thought it's about rendering the text encoder on CPU (or other GPU) to reduce the VRAM usage?

Juqowel commented 3 days ago

I thought it's about rendering the text encoder on CPU (or other GPU) to reduce the VRAM usage?

Yes, that's right.

I don't know why "Force/Set CLIP Device" uses 100% CPU.

HerMajestyDrMona commented 3 days ago

I don't know why "Force/Set CLIP Device" uses 100% CPU.

Isn't that exactly how it's supposed to work? It runs the T5 model on the CPU, therefore it's a bit slower (takes a few seconds depending on CPU if the prompt changes), and uses RAM instead of VRAM.

And with this "GPU_For_T5" extension, it doesn't seem to do that, so I assume it's still using my Nvidia GPU.

Juqowel commented 3 days ago

Isn't that exactly how it's supposed to work?

No.

You can check memory load process for second generation with changed prompt. With extension and without. Also you can see VRAM usage with sd1.5 and any SDXL.

HerMajestyDrMona commented 3 days ago

Isn't that exactly how it's supposed to work?

No.

You can check memory load process for second generation with changed prompt. With extension and without. Also you can see VRAM usage with sd1.5 and any SDXL.

Sorry, I think I'm too unclear. By "isn't that exactly how it's supposed to work" I meant the ComfyUI's "Force/Set Clip Device". It works that the full text encoder process is done on the CPU (or other selected card).

With this "GPU_for_T5" extension however it doesn't seem to do any work on the CPU after changing the prompt. It looks like the job is still being done on GPU (and VRAM).

So is there something that works incorrectly for me, or this extension is supposed to work differently from ComfyUIs' Clip Device selection? It's not supposed to actually render text encoding on the CPU, just to move models between VRAM and RAM more effectively?

Juqowel commented 3 days ago

I don't know why "Force/Set CLIP Device" uses 100% CPU. -> Isn't that exactly how it's supposed to work? -> No.

That's clear.

just to move models between VRAM and RAM more effectively?

No. It loads into RAM and stays there.

HerMajestyDrMona commented 2 days ago

No. It loads into RAM and stays there.

And still uses GPU to generate text? Something is weird for me.

If it doesn't use the CPU (doesn't show any short high CPU usage when changing prompt), then I assume it simply doesn't work on the CPU for me. So unlikely ComfyUI's stuff, this extension doesn't work for me on CPU.

The higher CPU usage for a few seconds is the characteristic of any AI model that runs on CPU instead of GPU.

Juqowel commented 2 days ago

And still uses GPU

It seems so.

HerMajestyDrMona commented 2 days ago

It seems so.

I wonder if you could test it when you have some free time? I'm not saying to fix it, just to test if it works fine for you on CPU with the newest version of forge?

Juqowel commented 2 days ago

I've already tested it. You don't have to worry, there shouldn't be any noticeable load on the CPU in Forge. No difference in the image with and without the extension.