googlecolab / colabtools

Python libraries for Google Colaboratory
Apache License 2.0
2.21k stars 728 forks source link

How to Remove "Can't initialize NVML" on Colab? #3392

Open BarteJJ opened 1 year ago

BarteJJ commented 1 year ago

Hello!

When I use T4 (Tesla 4) GPU on Colab, I get this error "/usr/local/lib/python3.8/dist-packages/torch/cuda/init.py:497: UserWarning: Can't initialize NVML warnings.warn("Can't initialize NVML")". It appears no matter am I using PC (Windows 11) or mobile device (Android). I use Chrome. How to remove it on Colab and is it even possible?

mayankmalik-colab commented 1 year ago

It seems like you are not using a GPU runtime.

For example: import torch torch.cuda.device_count() would through a warning Can't initialize NVML if not using a GPU runtime.

Please go to Runtime -> Change runtime type -> select GPU as hardware accelerator. Hopefully it should work.

BarteJJ commented 1 year ago

What is min repro notebook?Temat: Re: [googlecolab/colabtools] How to Remove "Can't initialize NVML" on Colab? (Issue #3392)Data: 2023-02-09 0:39Nadawca: "mayankmalik-colab" @.>Adresat: "googlecolab/colabtools" @.>; DW: "BarteJJ" @.>; "Author" @.>;

Can you please share a min repro notebook. I would be happy to debug.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

EvanWiederspan commented 1 year ago

Have you tried the Change Runtime Type command above?

BarteJJ commented 1 year ago

Not yet but... I suspect I already had GPU set up there then. Just need to be sure. Gonna tell you very soon...Temat: Re: [googlecolab/colabtools] How to Remove "Can't initialize NVML" on Colab? (Issue #3392)Data: 2023-02-09 18:16Nadawca: "Evan Wiederspan" @.>Adresat: "googlecolab/colabtools" @.>; DW: "BarteJJ" @.>; "Author" @.>;

Have you tried the Change Runtime Type command above?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

BarteJJ commented 1 year ago

I already checked those settings and unfortunately, as I suspected, runtime type already had set up GPU as a hardware accelerator. :(Temat: Re: [googlecolab/colabtools] How to Remove "Can't initialize NVML" on Colab? (Issue #3392)Data: 2023-02-09 18:16Nadawca: "Evan Wiederspan" @.>Adresat: "googlecolab/colabtools" @.>; DW: "BarteJJ" @.>; "Author" @.>;

Have you tried the Change Runtime Type command above?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

mayankmalik-colab commented 1 year ago

Can you share your notebook or minimal code that can help us reproduce the concern. I would be happy to take a look.

BarteJJ commented 1 year ago

It's from this:https://colab.research.google.com/drive/1X9dLe7_117rtuM3PJplJ9YMrZh9u1I0y?usp=sharingBefore you'll do anything there, make sure you clicked "File/Save a Copy in Drive". The problem appears after "Inference" (make sure GPU is checked there). This is a place an error I've mentioned about, appears.Temat: Re: [googlecolab/colabtools] How to Remove "Can't initialize NVML" on Colab? (Issue #3392)Data: 2023-02-09 20:37Nadawca: "mayankmalik-colab" @.>Adresat: "googlecolab/colabtools" @.>; DW: "BarteJJ" @.>; "Author" @.>;

Can you share your notebook or minimal code that can help us reproduce the concern. I would be happy to take a look.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>