Slow setup times on windows

LLukas22 commented 1 year ago

Setting up cuda-toolkit on a normal windows github actions runner can take >>15 min. I tried to speed this up by using the network installer and specify only the sub-packages i actually need but that also doesn't make any difference. The same setup on linux takes ~2:30 min also via the network method. Am i doing something wrong? What is the recommended way to install only specific packages on windows?

bvnp43 commented 11 months ago

ubuntu-20.04
cuda: 12.2.0
method: network
sub-packages: '["nvcc", "cudart"]'

28s

windows-latest
cuda: 12.2.0
method: network
sub-packages: '["nvcc", "cudart", "visual_studio_integration"]'

17m 36s

@Jimver

bullno1 commented 9 months ago

I tried messing around with the script: https://github.com/bullno1/cuda-toolkit/commits/master/. Newer installer, -n don't seem to make a difference.

The actual silent installation in my machine is very fast. If you look at the log,:when cached, even the local installer takes very little time to download. It just spends forever to install.

My guess is that it tries to extract the archive and the CI machine hardware is bad for that. Either spinning disk or throttled CPU.

The installer is also just an archive appended to an executable as most of them are. The official doc also mentions that you can extract it manually.

Maybe I'll attempt to do just that:

Extract the few packages that I actually need.
Repack in a different format.
Extract on CI.

bullno1 commented 9 months ago

It's only 6 minutes today: https://github.com/bullno1/hey/actions/runs/7282548943/job/19845086235#step:4:1

I have a feeling the issue is entirely with Github

rickardp commented 8 months ago

Was this ever looked into?

bvnp43 commented 7 months ago

It is visual_studio_integration component, that takes 20 minutes to install every time, other ones seems are pretty fast.

v0.2.14

rickardp commented 7 months ago

This is also what we found. Removing it speeds things up significantly but breaks some tooling. Is the same thing experienced on a Windows desktop? (Don't have one so cannot try)

Maybe it's possible to just unzip the installer files instead if the bug is in the installer itself?

bullno1 commented 7 months ago

It is visual_studio_integration component, that takes 20 minutes to install every time, other ones seems are pretty fast.

v0.2.14

Bummer, I think I need that or msbuild won't find the toolchain.

This is also what we found. Removing it speeds things up significantly but breaks some tooling. Is the same thing experienced on a Windows desktop? (Don't have one so cannot try)

Maybe it's possible to just unzip the installer files instead if the bug is in the installer itself?

The installer can be extracted, the problem is the transitive dependencies that you need to also install. AFAIK, there is very little doc on the structure of the archived files and where to extract them.

thewh1teagle commented 4 months ago

That's so slow! it takes me ~25 minutes only for setup nvidia step in my builds. Is there any workaround?

Looks like the installer itself is very heavy, I think that it tried to install the driver although I don't need it to compile with cuda, and also it does many checks before install it.

thewh1teagle commented 4 months ago

I had this problem with cuda setup taking forever, like 30 minutes. So I made a simple script setup_cuda.ps1, and now it finishes in just 2 minutes! ✨

ichinii commented 2 months ago

I had this problem with cuda setup taking forever, like 30 minutes. So I made a simple script setup_cuda.ps1, and now it finishes in just 2 minutes! ✨

Do you mind if we copy your script to our project?

thewh1teagle commented 2 months ago

I had this problem with cuda setup taking forever, like 30 minutes. So I made a simple script setup_cuda.ps1, and now it finishes in just 2 minutes! ✨

Do you mind if we copy your script to our project?

You can. Consider it as MIT license

Jimver / cuda-toolkit

Slow setup times on windows #253