search

conda-forge / miniforge

A conda-forge distribution.
https://conda-forge.org/miniforge
Other
5.76k stars 306 forks source link

Question about installation filesystem types and poor performance on NAS #507

Closed millsks closed 8 months ago

millsks commented 8 months ago

Comment:

We currently use miniforge for our project. We install everything on our local filesystems. No issue there. The problem that we are facing is when our CICD pipeline picks up the changes it does everything on NAS filesystems because each of their nodes is part of a cluster.

We have noticed dramatic decrease in performance when using conda/conda-build on these remote filesystems. If it is a basic build (e.g., 5 pkgs) it will finish in 15 minutes on NAS where if it was run on a local filesystem it runs in just a few minutes.

A big difference in time is seen when there are a lot of packages in the recipe (e.g. 180 pkgs). On a local filesystem it takes about 10 minutes where as if it is ran on NAS it can take upwards of 2 hours. I have noticed that a lot of time is taken resolving the dependencies and the urllib requests to the package repos. Is there anything that we can do to increase performance when running installs and builds on NAS?

hmaarrfk commented 8 months ago

install an ssd cache on you NAS. get 10Gb ethernet????

millsks commented 8 months ago

I can suggest that to them, but its part of an enterprise and it may not be a viable tactical option. possible down the road after all governance and management inputs are processed....yay corporate structure. :)

hmaarrfk commented 8 months ago

Try to test with an all SSD NAS, doesn't have to be big, but can help troubleshoot the problem.