beckernick
commented
3 years ago CUDA version mismatches would be the normal culprit, but I'm not sure how this would only show up intermittently. @jakirkham do you have any thoughts on this?
Open beckernick opened 3 years ago
beckernick
commented
3 years ago CUDA version mismatches would be the normal culprit, but I'm not sure how this would only show up intermittently. @jakirkham do you have any thoughts on this?
jakirkham
commented
3 years ago cc @anaruse (in case you have thoughts here 🙂)
anaruse
commented
3 years ago Hmm, it might have something to do with the issue below. https://github.com/cupy/cupy/issues/3935
VibhuJawa
commented
3 years ago Based on the comment they recommend setting the CUPY_GPU_MEMORY_LIMIT="90%" environment variable as they say there's no enough memory left to create cublas handler .
QQ : Is the cublas handler here supposed to use the existing rmm pool (because if it's not this may explain the pool growth we see while using SPACY)
cc: @beckernick
jakirkham
commented
3 years ago QQ : Is the cublas handler here supposed to use the existing rmm pool (because if it's not this may explain the pool growth we see while using SPACY)
I don't think so (meaning it shouldn't use the pool).
Edit: To expand on this, we would need an API in cuBLAS (that CuPy would then use), which would allow us to specify a chunk of memory to use for the initialization.
jakirkham
commented
3 years ago Is there a way for us to trigger cuBLAS initialization early? If so, maybe we can do this as part of Dask-CUDA startup.
beckernick
commented
3 years ago Thanks for the pointer @anaruse @jakirkham . Let's look into triggering this early and/or perhaps reserving memory.
beckernick
commented
3 years ago @anaruse @jakirkham , do you think there is any downside/risk of initializing a handle to the cuBLAS library context and then essentially "throwing it away" before we do anything else on the cluster?
I.e., running something like this on every Dask worker?
def init_cublas():
from cupy.cuda import cublas
cublas.create() # allocates 64MB of GPU memory and returns a handle
return None
client.run(init_cublas)Initializing the cuBLAS context beforehand wouldn't necessarily change the allocation dynamics of the workload that triggers RMM to grow the pool to just at the edge of total capacity. Without visibility into that chunk of memory, this might still be a risk, right?
Perhaps in combination with CUPY_GPU_MEMORY_LIMIT we may be able to avoid it, but it's not immediately clear to me how that would work.
jakirkham
commented
3 years ago Idk about creating cuBLAS handle like that. The library may expect us to do cleanup. Not sure what happens if we don't do that cleanup.
That said, maybe we can do some warmup step (like matrix multiplication), which would get CuPy to initialize cuBLAS. Admittedly that's a bit hacky, but perhaps workable.
@anaruse may have a better suggestion
jakirkham
commented
3 years ago I poked around the CuPy code and think something like this might work. Should add CuPy takes care of the cleanup of the handle in this case
In [1]: import cupy
In [2]: cupy.cuda.device.get_cublas_handle()
Out[2]: 94517515719584
In [3]: cupy.cuda.device.get_cublas_handle()
Out[3]: 94517515719584The first run is a bit slow (as it allocates the handle), but the second one is a bit faster (as it is cached). Note the pointer returned as a Python int is the same in both cases
We could call this with client.run or similar to make sure this is setup on all workers
jakirkham
commented
3 years ago Mentioned in an edit in the RMM thread, but it seems spaCy just uses CuPy for cuBLAS and doesn't use cuBLAS directly. So I think that initialization step should be sufficient
This is the intermittent
CUBLAS_STATUS_NOT_INITIALIZEDerror that I thought was in Q28 @VibhuJawa