Library libcudart.so.11.0 not found in jupyter-notebook

[hypercube999]

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.] Install juju Install microk8s version 1.24.17 Install kubeflow (juju deploy kubeflow --trust --channel=1.7/stable) Create a jupyter-notebook with any image designed to work with gpu, e.g. charmedkubeflow/jupyter-tensorflow-cuda-full:v1.7.0_20.04_1 Start the notebook and run a program in it that uses a video card, for example a python program that trains a neural network using tensorflow. Jupyter show error "Could not load dynamic library 'libcudart.so.11.0' <...> No such file or directory"

What did you expect to happen: The program works using gpu.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.] On another similar deployment done in late August everything works fine

Environment:

• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): 1.7/stable
• Kubernetes platform: (e.g. minikube) microk8s
• Kubernetes version: (use kubectl version): 1.24.17
• OS (e.g. from /etc/os-release): ubuntu server 20.04

[thesuperzapper]

@hypercube999 what happens when you use the updated images from Kubeflow 1.8 (which can be used with older Kubeflow versions too).

For example: kubeflownotebookswg/jupyter-tensorflow-cuda-full:v1.8.0

[andreyvelich]

/transfer notebooks