search

LLNL / lbann

Livermore Big Artificial Neural Network Toolkit
http://software.llnl.gov/lbann/
Other
223 stars 79 forks source link

cuTENSOR error (CUTENSOR_STATUS_NOT_SUPPORTED) #2463

Closed jvwilliams23 closed 1 month ago

jvwilliams23 commented 2 months ago

Hi

As I mentioned on Wednesday, we are getting an CUTENSOR_STATUS_NOT_SUPPORTED when doing lbann.Tessellate (also with lbann.MultiDimReduction mentioned previously #2429) with distconv.

I found that the issue only occurs when mini-batch-size = 1 (or I guess, when mini-batch size < number of gpus). I put the relevant lines from the cuTENSOR log below.

The last entry in the “extent”, which appears to be batch dimension, is split across each gpu (I thought it would be copied). When there is only a single sample, this causes process 0 to have size = 1 in the batch dimension, and process 1 to have size = 0.

[2024-07-12 12:39:08][cuTENSOR][2491675][Api][cutensorInitTensorDescriptor] handle=0X7F84A24F35E0 desc=0X7FFF83FFACE0 numModes=4 extent=[8,8,512,0] stride=[1,8,64,32768] dataType=0 op=1
[2024-07-12 12:39:08][cuTENSOR][2491675][Error][cutensorInitTensorDescriptor] Extent (0) must be > 0.
[2024-07-12 12:39:08][cuTENSOR][2491675][Api][cutensorGetErrorString] error=15
[2024-07-12 12:39:08][cuTENSOR][2491674][Api][cutensorInitTensorDescriptor] handle=0X7FB1558F35E0 desc=0X7FFC7845B6C0 numModes=4 extent=[8,8,512,1] stride=[1,8,64,32768] dataType=0 op=1

Josh

tbennun commented 1 month ago

This should be fixed now