cjnolet
commented
4 years ago Logistic regression seems to throw an error before the algorithm even executes and it seems to be from CuPy:
a = np.random.random((1000000, 2100))
from cuml.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(a, a[:,0])Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/raid/cnolet/miniconda3/envs/rapidgenomics2/lib/python3.7/site-packages/cuml/common/memory_utils.py", line 56, in cupy_rmm_wrapper
return func(*args, **kwargs)
File "cuml/linear_model/logistic_regression.pyx", line 262, in cuml.linear_model.logistic_regression.LogisticRegression.fit
File "/raid/cnolet/miniconda3/envs/rapidgenomics2/lib/python3.7/site-packages/cupy/manipulation/add_remove.py", line 74, in unique
mask = cupy.empty(aux.shape, dtype=cupy.bool_)
File "/raid/cnolet/miniconda3/envs/rapidgenomics2/lib/python3.7/site-packages/cupy/creation/basic.py", line 22, in empty
return cupy.ndarray(shape, dtype, order=order)
File "cupy/core/core.pyx", line 137, in cupy.core.core.ndarray.__init__
File "cupy/cuda/memory.pyx", line 569, in cupy.cuda.memory.alloc
File "/raid/cnolet/miniconda3/envs/rapidgenomics2/lib/python3.7/site-packages/rmm/rmm.py", line 347, in rmm_cupy_allocator
buf = librmm.device_buffer.DeviceBuffer(size=nbytes)
File "rmm/_lib/device_buffer.pyx", line 79, in rmm._lib.device_buffer.DeviceBuffer.__cinit__
MemoryError: std::bad_alloc: CUDA error at: /conda/conda-bld/librmm_1591196517113/work/include/rmm/mr/device/managed_memory_resource.hpp70: cudaErrorIllegalAddress an illegal memory access was encountered
github-actions[bot]
wphicks
Below is a reproducible example on the 0.15 nightly of cuML w/ CUDA 10.2.
There are several estimators that fail with either
illegal memory access was encounteredorinvalid configuration argument, both of which seem to indicate an integer overflow might be occurring when representing the size of the underlying array.Here's the exception
In the above example, I used a size of
1Mx5000, which is >>2^31number of samples. I also tried with1Mx2500, which is also >2^31.Just to rule out the possibility that this error only occurs in the case of oversubscribing the GPU memory, I tried with
1Mx2100, (which is<2^31but still requires>32gbof GPU memory to train):As a result of the behavior outlined above, I have a very strong suspicion the are integers being used to represent array sizes that should be promoted to
size_t. This should also affectTruncatedSVD.Keeping a list to track the places where this occurs so far: