search

StatCan / aaw-kubeflow-containers

Containers built to be used with Kubeflow for Data Science
Other
24 stars 22 forks source link

JupyterLab-CPU image problem with Conda #426

Closed asolis closed 1 year ago

asolis commented 1 year ago

We are currently facing multiple issues with Conda. In particularly, we are unable to recreate conda environments from a environment.yml file, we are mostly using conda-forge. I have notice that the image being loaded into the pod isn't carrying all the files as in the master branch of this repository. The .condarc file that was push in the pull request #420 is not showing when creating new notebooks. We have seen different images allocated to a new notebook, but I'm not able to replicate this behavior at this point.

We are also experiencing problems with "doble environments" (python-base-venv)(base) is showing in the prompt. Creating conflicts when installing some packages.

@bryanpaget is currently working with Johan Fernandes (DScD) in getting access to the AAW namespace "dsd-aeas-hydra" to inspect the behavior regarding multiple Conda environments being loaded .

I'm cc'ing @Collinbrown95 because we worked together on the pull request #420, the changes being made in this request are not being present on the images being loaded into the notebook pods.

asolis commented 1 year ago

Cc: @johanfernandes

Jose-Matsuda commented 1 year ago

We had initially done the double environments to avoid users from accidentally messing with their notebooks. See https://github.com/StatCan/aaw-kubeflow-containers/issues/417 and https://github.com/StatCan/aaw-kubeflow-containers/pull/419

I can revert / update our images to not have this requirement again and take another look at it as I wasn't aware that this would become a problem

Jose-Matsuda commented 1 year ago

as for the .condarc file I actually don't remember know how that plays when you place it inside the dockerfile, i think once you attach persistent storage it will go away ill check right now I may move it to the startup-custom.sh

image

(this is on a v1 jupyterlab-cpu notebook with no volume, once you attach one it goes away / gives it a clean slate)

asolis commented 1 year ago

@Jose-Matsuda . the reason for the .condarc file is to create Conda permanent environments. By default the conda environment location is inside /opt/conda which is ephemeral in the pod. The configuration inside the /home/joyvan/.condarc allows to specify a new location under /home/joyvan/ which is attached as a volume and not ephemeral. Any new environment created with this setting enabled will be stored in the volume (under the specified location). Stopping and resuming the pod will not incur in re-generating the Conda environments manually. Each notebook we created in our namespace was missing the .condarc file, which is a weird giving that it's showing in yours.

asolis commented 1 year ago

*note: we were using v1 jupyterlab-cpu as the base image too

Jose-Matsuda commented 1 year ago

@asolis yeah the reason it shows up is because i didnt mount a volume; this came up when i was initially looking into persisting conda envs in our images with things already installed.

In the image, when you install packages or whatever in a path that is eventually used by persistent storage, if you mount persistent storage it will go away.

I'm working on a really quick fix to this and may want you to take a look at the PR to see if everything makes sense and then can merge it and it will take an hour for the images to build and push to our acr

Jose-Matsuda commented 1 year ago

Image should be available in an hour or so, should be able to just restart your v1 notebook and python venv restrictions will be gone.

If it's not solved can re-open it