Can't use UMAP after having been using it for a while

[effyli]

I have been using bertopic for training topic models during the past months, and it worked fine. However, since last week, I have been running into issues with making it run on the clusters from my university. I kept getting errors as follows, and I couldn't find any similar issues here. It seems like a cuBLAS version is incompatible but I am not sure how I can fix it. This error is only thrown after running the script for around 40 minutes. That's why I also reduced the n_neighbors parameter in the UMAP function, which was running fine before.

Any suggestions will be appreciated!

Traceback (most recent call last): File "/var/scratch/xli3/workspace/dstm/dstm/bert-topic.py", line 104, in topics, probs = topic_model.fit_transform(corpus) File "/var/scratch/xli3/miniconda3/envs/dstm2/lib/python3.10/site-packages/bertopic/_bertopic.py", line 386, in fit_transform umap_embeddings = self._reduce_dimensionality(embeddings, y) File "/var/scratch/xli3/miniconda3/envs/dstm2/lib/python3.10/site-packages/bertopic/_bertopic.py", line 3183, in _reduce_dimensionality self.umap_model.fit(embeddings, y=y) File "/var/scratch/xli3/miniconda3/envs/dstm2/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 188, in wrapper ret = func(*args, kwargs) File "/var/scratch/xli3/miniconda3/envs/dstm2/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 393, in dispatch return self.dispatch_func(func_name, gpu_func, *args, *kwargs) File "/var/scratch/xli3/miniconda3/envs/dstm2/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 190, in wrapper return func(args, kwargs) File "base.pyx", line 674, in cuml.internals.base.UniversalBase.dispatch_func File "umap.pyx", line 603, in cuml.manifold.umap.UMAP.fit RuntimeError: cuBLAS error encountered at: file=/__w/cuml/cuml/python/_skbuild/linux-x86_64-3.10/cmake-build/_deps/raft-src/cpp/include/raft/core/resource/cublas_handle.hpp line=75: call='cublasSetStream(ret, get_cuda_stream(res))', Reason=1:CUBLAS_STATUS_NOT_INITIALIZED

[MaartenGr]

Thanks for sharing this issue. You mention that it is not working anymore. Did something change between when it was working and now? For instance, do the environments that you use all have the same versions with respect to BERTopic, its dependencies, python, CUDA, etc.?

[effyli]

Thanks for the reply! I am using conda to manage my environments. However, I also tried to reinstall everything and created a new environment but the error still shows in this new environment. As for CUDA, I was trying to load the same module and I haven't been informed of any changes on that. Could there be any version issue with the new Bertopic? Because I think I am installing the new version in this newly created environment.

[effyli]

The problem is still there... Any suggestions?

Thanks for sharing this issue. You mention that it is not working anymore. Did something change between when it was working and now? For instance, do the environments that you use all have the same versions with respect to BERTopic, its dependencies, python, CUDA, etc.?

[MaartenGr]

Could there be any version issue with the new Bertopic? Because I think I am installing the new version in this newly created environment.

If you think this might be the culprit, then it is unlikely there was version control with respect to all dependencies. As a result, I believe that the environment that you used previously was not the same as you are using currently. Any changes between them might affect functionality. Having said that, I do not think this is related to BERTopic but to cuML. Are you making sure you are installing the exact same version of cuML as you did previously? Also, are you using the exact same GPU with the same CUDA version as you did previously? From a quick google search, it seems to relate to incompatibility between your GPU and CUDA.