search

space-nuko / a1111-stable-diffusion-webui-vram-estimator

Show estimated VRAM usage for generation configs
MIT License
107 stars 11 forks source link

Benchmark always results in IndexError when finished #1

Open jwvanderbeck opened 1 year ago

jwvanderbeck commented 1 year ago

benchmark error: CUDA out of memory. Tried to allocate 7.15 GiB (GPU 0; 24.00 GiB total capacity; 10.23 GiB already allocated; 2.73 GiB free; 17.70 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF Traceback (most recent call last): File "D:\Developer\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 337, in run_predict output = await app.get_blocks().process_api( File "D:\Developer\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1015, in process_api result = await self.call_function( File "D:\Developer\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 833, in call_function prediction = await anyio.to_thread.run_sync( File "D:\Developer\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "D:\Developer\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "D:\Developer\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 867, in run result = context.run(func, *args) File "D:\Developer\stable-diffusion-webui\extensions\a1111-stable-diffusion-webui-vram-estimator\scripts\vram_estimator.py", line 201, in run_benchmark curves[k] = VRAMCurve(v) File "D:\Developer\stable-diffusion-webui\extensions\a1111-stable-diffusion-webui-vram-estimator\scripts\vramestimator.py", line 71, in init A = np.c[np.ones(data.shape[0]), data[:,:2], data[:,0]**2, np.prod(data[:,:2], axis=1), \ IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

space-nuko commented 1 year ago

Is there a stats.json file in the extension's folder? If so what are its contents?

jwvanderbeck commented 1 year ago

Pulled it into vscode and looks to be valid json but I only looked real quick. It has a lot of data though so attaching it here rather than pasting it stats.zip

space-nuko commented 1 year ago

Seems like it's not saving any results for the latent benchmark for some reason, it's just an empty array. The Latent sampler in highres fix for txt2img works right? Were there any errors printed during the benchmark?

space-nuko commented 1 year ago

Oh wait I see, it ran out of memory during the latent benchmark. I guess that should be made clearer. Maybe reduce the maximum size/batch count to the amount you can use with Latent upscalers?

jwvanderbeck commented 1 year ago

I don't normally use it but it does work (I've used it by accident once or twice). I don't recall any errors other than the one I posted.

I did originally try running a benchmark with different settings, with a 2048 size and only 2 count which failed pretty quick with the error which is when I then reset the settings to the defaults thinking it might be a bug with only 2 count, but then got the same error.

I don't know if that had anything to do with it.

I will try deleting the stats.json file and rerunning with the default settings

jwvanderbeck commented 1 year ago

Well I deleted the file and reran it with the default settings and it worked, so it must be as you say when I tried to run it at 2048 and then the file got corrupted?

That said isn't the whole point here that this extension goes until it runs out of memory? Is there something special about this stage that you can't detect the oom condition?

space-nuko commented 1 year ago

I tried doing that earlier, the problem is sometimes it doesn't throw an OOM exception if you just barely cross the VRAM limit and instead the program just hangs indefinitely, even all the way overnight