Google Colab notebook running out of disk space installing tf-nightly from clu

404pr commented 2 years ago

Issue

This notebook runs out of disk space installing tf-nightly when running the Setup Environment cell.

Details

I poked around a bit, and it looks like as part of the t5x installation, clu also gets installed. During the clu installation, it looks like tf-nightly attempts to get installed twice. The first time around it grabs the most recent version, tf_nightly-2.10.0.dev20220521-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl, as seen here

The second time around, it tries to grab every version (including back to 2.9.0) from https://pypi.org/project/tf-nightly/#history, as seen here , until it eventually hits ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device

I'm not quite sure what's causing the second installation to trigger. The rest of the installation seems to go okay, but running Imports and Definitions fails out when it can't find clu

Notes

These errors came from running !pip install git+https://github.com/google/CommonLoopUtils#egg=clu , which I got from the t5x repo . If it makes more sense to open an issue there, I can do that -- I opened one here as it happened when I was running the demo notebook
This has run for me as recently as April 26, 2022. I'm not sure if this issue was still happening, but I was just not running out of disk space on Colab because there were fewer tf-nightly versions (they're each ~500MB)

JJOL commented 2 years ago

[Solved] I'm facing this problem as well. Last time I was able to correctly run a Colab Instance of this notebook was on April 24. Maybe by manually installing clu and t5x with their heads pointing to the last commit before April 24 the script could run.

I provide a copy of MT3's notebook with the above changes which fixes the issue.

Does anyone know if non colab environments are safe from this issue?

404pr commented 2 years ago

Oh @JJOL , nice, I shoulda thought of testing that out. I can confirm that your changes got me through the installation successfully.

I actually just gave this a whirl on my local,

pip install git+https://github.com/google/CommonLoopUtils#egg=clu and the only tf-nightly version that got installed was 2.10.0.dev20220528.

pip 22.0.4  
python 3.8.2 
macOS Catalina 10.15.77
Intel chip

So yeah, that's kinda interesting, it does seem that it may be Google Colab specific, although I'm not sure why.

iansimon commented 2 years ago

This seems to be fixed now.

magenta / mt3