RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

Running on Ubuntu 22.04 using conda with python 3.7 and a Titan X GPU with CUDA 11
conda create --name galactica python=3.7
conda activate galactica
pip install git+https://github.com/paperswithcode/galai
python3.7
>>> import galai as gal
>>> model=gal.load_model("mini", num_gpus=1)
>>> model.generate("Scaled dot product attention:\n\n\\[")

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/galai/model.py", line 140, in generate
    output_hidden_states=True
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/transformers/generation_utils.py", line 1499, in generate
    **model_kwargs,
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/transformers/generation_utils.py", line 2237, in greedy_search
    output_hidden_states=output_hidden_states,
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/galai/architecture.py", line 974, in forward
    return_dict=return_dict,
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/galai/architecture.py", line 732, in forward
    use_cache=use_cache,
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/accelerate/hooks.py", line 156, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/galai/architecture.py", line 333, in forward
    output_attentions=output_attentions,
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/accelerate/hooks.py", line 156, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/galai/architecture.py", line 178, in forward
    query_states = self.q_proj(hidden_states) * self.scaling
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/accelerate/hooks.py", line 156, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
paperswithcode / galai

RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)` #35
