aws / studio-lab-examples

Example notebooks for working with SageMaker Studio Lab. Sign up for an account at the link below!
https://studiolab.sagemaker.aws
Apache License 2.0
655 stars 193 forks source link

inability to reimage SageMaker Studio Lab instance to get the space back #167

Closed boris-korotkov closed 2 years ago

boris-korotkov commented 2 years ago

Hello,

After I tried to build a Conda environment using mlu-tab.yml I was ran out of space with no environment created. After I deleted all files from my home folder I still had 95% of my space used. There is no way to "reimage" my Studio Lab instance and get back the initial 30Gb of space.

I followed the AWS Machine Learning University course and cloned the examples for Tabular data course: [(https://github.com/aws-samples/aws-machine-learning-university-accelerated-tab)]

After that I was stupid enough to try creating the Conda environment using the mlu-tab.yml file. the environment creation ate all my space available and creation was failed. Currently I have 95% space usage of my /home/studio-lab-user folder with no files in it.

How can I reimage SageMaker Studio Lab instance to get the space back or uninstall all libraries installed by creating the Conda environment?

OS: Windows 10 Browser: Chrome 107.0.5304.107

space issue1 space issue2

boris-korotkov commented 2 years ago

Sorry, I haven't done enough research in the existing issue. In fact, there is a solution in issue#42. While a simple button like reset the instance would help, the removing .conda folder helped to release the space.

icoxfog417 commented 2 years ago

Thank you for your asking the question and self-service solving! I am glad to hear existing issue (#42 ) worked for you.

If you need the reset feature, please up vote ( #75 ) :) .

nollok-official commented 2 years ago

Hello you can follow the following steps and run the codes provided herewith in a terminal (you can open a terminal after ) :

After you have done all of this stop your project from sagemaker main dashboard. Open a new project. Now you will get a fresh new notebook with almost all the space available. (You can check available space by running "df -h", take a look at Avail row beside "/home/studio-lab-user").

Note that this process frees us 'almsot' all he space, you may still have 2-3 gb of occupies space. Still d'ont know how to fix it

#1.  switch to base conda environment
conda activate base

#2. List all conda environments available in your account
conda  list envs

#Take a look at all the conda environments  available
#3. Delete all the conda environments except environment named 'base'
conda remove -n "env_name" --all

#4.Delete all subdirectories from user folder
rm -rf /home/studio-lab-user/*

#Restart project/runtime
boris-korotkov commented 2 years ago

Thank you for your response @yeasin-arafat-rafio ! The issue has already been closed. The problem was that new environment was not created because available space was not sufficient to install all packages. You can reproduce the issue by cloning MLU course from Github with enabled option to create environment.yml file. aws-machine-learning-university-accelerated-tab . Then you can use that yml file to create a conda environment required for the course. Since it was the default course suggested on the title page of Sagemaker Studio Lab I thought it was tested and safe to use, but I was wrong.

nollok-official commented 2 years ago

@boris-korotkov Hey, how did you reset the instance. I haven't tried that cuz I couldn't figure out how you did that. Did you like restart the instance from the main dashboard by stopping the runtime and again starting the runtime