Chia-Network / bladebit

A high-performance k32-only, Chia (XCH) plotter supporting in-RAM and disk-based plotting
Apache License 2.0
339 stars 109 forks source link

cudaErrorMemoryAllocation in WSL2 with 1080Ti 11GB #314

Open spleen911 opened 1 year ago

spleen911 commented 1 year ago

My plotter doesn't have TPM2 and E5-2660v4 isn't technically supported by Windows 11 so for now I'm staying on Windows 10 which I know has issues generating valid cuda plots. That said, I'm trying to use WSL and bladebit-cuda-v3.0.0-alpha3-ubuntu-x86-64.tar.gz, but I get a cudaErrorMemoryAllocation when it tries to allocate buffers.

I have 512GB and ramplot in Windows without issue. Windows cudaplot also works, but of course the plots won't pass proofs. I'll be honest, I don't know anything about Ubuntu and did a basic WSL install with apt update && apt upgrade so I am hoping I just need to set something additional. Below are nvidia-smi and bladebit_cuda output.

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 531.18       CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 960          On | 00000000:03:00.0 Off |                  N/A |
|  0%   41C    P8               15W / 128W|    246MiB /  2048MiB |      3%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce GTX 1080 Ti      On | 00000000:A1:00.0 Off |                  N/A |
| 23%   32C    P8               10W / 250W|     15MiB / 11264MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

Bladebit Chia Plotter
Version      : 3.0.0-alpha3
Git Commit   : 816388914720c64480665519d52fd075ca528063
Compiled With: gcc 9.4.0

[Global Plotting Config]
 Will create 1 plots.
 Thread count          : 56
 Warm start enabled    : false
 NUMA disabled         : false
 CPU affinity disabled : false
 Farmer public key     : redacted
 Pool contract address : redacted
 Benchmark mode        : disabled

[Bladebit CUDA Plotter]
Selected cuda device 0 : NVIDIA GeForce GTX 1080 Ti
 CUDA Compute Capability   : 6.1
 SM count                  : 28
 Max blocks per SM         : 32
 Max threads per SM        : 2048
 Async Engine Count        : 5
 L2 cache size             : 2.75 MB
 L2 persist cache max size : 0.00 MB
 Stack Size                : 1.00 KB
 Memory:
  Total                    : 11.00 GB
  Free                     : 9.97 GB

Allocating buffers (this may take a few seconds)...
Kernel RAM required       : 90240524288  bytes ( 86060.07  MiB or 84.04  GiB )
Intermediate RAM required : 2999001088   bytes ( 2860.07   MiB or 2.79   GiB )
Host RAM required         : 168443248640 bytes ( 160640.00 MiB or 156.88 GiB )
Total Host RAM required   : 258683772928 bytes ( 246700.07 MiB or 240.92 GiB )
GPU RAM required          : 6139441152   bytes ( 5855.03   MiB or 5.72   GiB )
Allocating buffers
CUDA error: 2 (0x2 ) cudaErrorMemoryAllocation : out of memory

*** Panic!!! *** Fatal Error:
CUDA error cudaErrorMemoryAllocation : out of memory.
./bladebit_cuda(+0xe180b)[0x55b440b2980b]
./bladebit_cuda(+0xe0fef)[0x55b440b28fef]
./bladebit_cuda(+0x1b71a)[0x55b440a6371a]
./bladebit_cuda(+0x1bd34)[0x55b440a63d34]
./bladebit_cuda(+0x180c7)[0x55b440a600c7]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fa99d2abd90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fa99d2abe40]
./bladebit_cuda(+0x1984e)[0x55b440a6184e]
LeroyINC commented 1 year ago

Do both your GPU's have enough on-board memory to run GPU plotting? I believe you need 8GB of Ram on each card.

or wonder if the 1st GPU is interfering with the plotter and maybe it's using the wrong one?

spleen911 commented 1 year ago

I thought similar even though it says it is only using the 1080Ti (I also forced selected GPU using -d=# arg).

After posting, I physically removed the GTX960, but got the same result with only one GPU installed.

spleen911 commented 1 year ago

Good news: I upgraded to WIN11 and I can generate good plots now with alpha3. I have 162TB of new drives I can now plot via cudaplot vs ramplot

Bad news: WSL2 still gives me a cuda "out of memory" but so does gigahorse plotter. Something is definitely goofy with the WSL2 installation.