search

charlesfrye / dreambooth-modal

run dreambooth training on modal
21 stars 3 forks source link

BlockingIOError: [Errno 35] write could not complete without blocking #9

Open Klaudioz opened 1 year ago

Klaudioz commented 1 year ago

Hello, I've followed these steps:

git clone https://github.com/charlesfrye/dreambooth-modal.git
cd dreambooth-modal

mv .env.secrets.example .env.secrets
# Add the keys to .env.secrets
make environment
modal token new # Needed registered account
make model

And I've got the following error: BlockingIOError: [Errno 35] write could not complete without blocking after running make model

This is the entire log:

 ~/p/S/m/dreambooth-modal> make model                                                                  22.5s  Mon Nov 21 21:58:55 2022
pip install -r requirements-local.txt
Requirement already satisfied: python-dotenv in /usr/local/lib/python3.10/site-packages (from -r requirements-local.txt (line 1)) (0.20.0)
Requirement already satisfied: modal-client in /usr/local/lib/python3.10/site-packages (from -r requirements-local.txt (line 2)) (0.40.345)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (3.7.4)
Requirement already satisfied: types-toml in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (0.10.8.1)
Requirement already satisfied: protobuf<5.0,>=3.19 in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (4.21.9)
Requirement already satisfied: asgiref in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (3.5.2)
Requirement already satisfied: rich>=12.0.0 in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (12.6.0)
Requirement already satisfied: typer==0.6.1 in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (0.6.1)
Requirement already satisfied: watchfiles in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (0.18.1)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (5.0.0)
Requirement already satisfied: synchronicity>=0.2.13 in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (0.2.13)
Requirement already satisfied: sentry-sdk in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (1.11.0)
Requirement already satisfied: certifi in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (2021.10.8)
Requirement already satisfied: types-certifi in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (2021.10.8.3)
Requirement already satisfied: grpclib==0.4.3 in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (0.4.3)
Requirement already satisfied: toml in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (0.10.2)
Requirement already satisfied: aiostream in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (0.4.5)
Requirement already satisfied: tblib>=1.7.0 in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (1.7.0)
Requirement already satisfied: cloudpickle<2.1.0,>=2.0.0 in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (2.0.0)
Requirement already satisfied: fastapi in /usr/local/lib/python3.10/site-packages (from modal-client->-r requirements-local.txt (line 2)) (0.87.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.10/site-packages (from grpclib==0.4.3->modal-client->-r requirements-local.txt (line 2)) (4.5.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.10/site-packages (from grpclib==0.4.3->modal-client->-r requirements-local.txt (line 2)) (4.1.0)
Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.10/site-packages (from typer==0.6.1->modal-client->-r requirements-local.txt (line 2)) (8.1.3)
Requirement already satisfied: commonmark<0.10.0,>=0.9.0 in /usr/local/lib/python3.10/site-packages (from rich>=12.0.0->modal-client->-r requirements-local.txt (line 2)) (0.9.1)
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in /usr/local/lib/python3.10/site-packages (from rich>=12.0.0->modal-client->-r requirements-local.txt (line 2)) (2.13.0)
Requirement already satisfied: typing-extensions>=3.6.5 in /usr/local/lib/python3.10/site-packages (from aiohttp->modal-client->-r requirements-local.txt (line 2)) (4.3.0)
Requirement already satisfied: chardet<4.0,>=2.0 in /usr/local/lib/python3.10/site-packages (from aiohttp->modal-client->-r requirements-local.txt (line 2)) (3.0.4)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/site-packages (from aiohttp->modal-client->-r requirements-local.txt (line 2)) (1.3.0)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/site-packages (from aiohttp->modal-client->-r requirements-local.txt (line 2)) (19.1.0)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /usr/local/lib/python3.10/site-packages (from aiohttp->modal-client->-r requirements-local.txt (line 2)) (3.0.1)
Requirement already satisfied: starlette==0.21.0 in /usr/local/lib/python3.10/site-packages (from fastapi->modal-client->-r requirements-local.txt (line 2)) (0.21.0)
Requirement already satisfied: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 in /usr/local/lib/python3.10/site-packages (from fastapi->modal-client->-r requirements-local.txt (line 2)) (1.10.2)
Requirement already satisfied: anyio<5,>=3.4.0 in /usr/local/lib/python3.10/site-packages (from starlette==0.21.0->fastapi->modal-client->-r requirements-local.txt (line 2)) (3.6.2)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/site-packages (from importlib-metadata->modal-client->-r requirements-local.txt (line 2)) (3.10.0)
Requirement already satisfied: urllib3>=1.26.11 in /usr/local/lib/python3.10/site-packages (from sentry-sdk->modal-client->-r requirements-local.txt (line 2)) (1.26.12)
Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette==0.21.0->fastapi->modal-client->-r requirements-local.txt (line 2)) (1.3.0)
Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette==0.21.0->fastapi->modal-client->-r requirements-local.txt (line 2)) (3.3)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.10/site-packages (from h2<5,>=3.1.0->grpclib==0.4.3->modal-client->-r requirements-local.txt (line 2)) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.10/site-packages (from h2<5,>=3.1.0->grpclib==0.4.3->modal-client->-r requirements-local.txt (line 2)) (4.0.0)

[notice] A new release of pip available: 22.3 -> 22.3.1
[notice] To update, run: python3.10 -m pip install --upgrade pip
modal app deploy wandb_secret.py
modal app deploy hf_secret.py
A100=1 MODAL_GPU=1 modal app run run.py --function-name train
Building image im-buzjgafjcddrzhojrbxaxudrxlnh

=> Step 0: FROM base

=> Step 1: RUN python -m pip install 'diffusers>==0.5.0' accelerate torchvision 'gradio~=3.6' 'transformers>=4.21.0' ftfy tensorboard modelcards smart_open requests climage wandb python-dotenv
Collecting diffusers>==0.5.0
  Downloading diffusers-0.7.2-py3-none-any.whl (304 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 304.9/304.9 kB 79.4 MB/s eta 0:00:00
Collecting accelerate
  Downloading accelerate-0.14.0-py3-none-any.whl (175 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 176.0/176.0 kB 449.7 MB/s eta 0:00:00
Collecting torchvision
  Downloading torchvision-0.14.0-cp310-cp310-manylinux1_x86_64.whl (24.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.3/24.3 MB 461.4 MB/s eta 0:00:00
Collecting gradio~=3.6
  Downloading gradio-3.10.1-py3-none-any.whl (11.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.6/11.6 MB 508.9 MB/s eta 0:00:00
Collecting transformers>=4.21.0
  Downloading transformers-4.24.0-py3-none-any.whl (5.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 328.1 MB/s eta 0:00:00
Collecting ftfy
  Downloading ftfy-6.1.1-py3-none-any.whl (53 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.1/53.1 kB 369.3 MB/s eta 0:00:00
Collecting tensorboard
  Downloading tensorboard-2.11.0-py3-none-any.whl (6.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 484.1 MB/s eta 0:00:00
Collecting modelcards
  Downloading modelcards-0.1.6-py3-none-any.whl (12 kB)
Collecting smart_open
  Downloading smart_open-6.2.0-py3-none-any.whl (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.6/58.6 kB 379.9 MB/s eta 0:00:00
Collecting requests
  Downloading requests-2.28.1-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 402.6 MB/s eta 0:00:00
Collecting climage
  Downloading climage-0.1.3-py3-none-any.whl (19 kB)
Collecting wandb
  Downloading wandb-0.13.5-py2.py3-none-any.whl (1.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 481.5 MB/s eta 0:00:00
Collecting python-dotenv
  Downloading python_dotenv-0.21.0-py3-none-any.whl (18 kB)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.10/site-packages (from diffusers>==0.5.0) (4.8.1)
Collecting regex!=2019.12.17
  Downloading regex-2022.10.31-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (770 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 770.5/770.5 kB 467.5 MB/s eta 0:00:00
Collecting Pillow<10.0
  Downloading Pillow-9.3.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 212.7 MB/s eta 0:00:00
Collecting filelock
  Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/site-packages (from diffusers>==0.5.0) (1.23.4)
Collecting huggingface-hub>=0.10.0
  Downloading huggingface_hub-0.11.0-py3-none-any.whl (182 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 182.1/182.1 kB 447.7 MB/s eta 0:00:00
Collecting pyyaml
  Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 682.2/682.2 kB 490.7 MB/s eta 0:00:00
Collecting torch>=1.4.0
  Downloading torch-1.13.0-cp310-cp310-manylinux1_x86_64.whl (890.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 890.1/890.1 MB 455.1 MB/s eta 0:00:00
Collecting psutil
  Downloading psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 280.2/280.2 kB 469.1 MB/s eta 0:00:00
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/site-packages (from accelerate) (21.3)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/site-packages (from torchvision) (4.4.0)
Collecting nvidia-cudnn-cu11==8.5.0.96
  Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 557.1/557.1 MB 323.3 MB/s eta 0:00:00
Collecting nvidia-cuda-runtime-cu11==11.7.99
  Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 849.3/849.3 kB 478.7 MB/s eta 0:00:00
Collecting nvidia-cuda-nvrtc-cu11==11.7.99
  Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.0/21.0 MB 339.8 MB/s eta 0:00:00
Collecting nvidia-cublas-cu11==11.10.3.66
  Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 317.1/317.1 MB 481.1 MB/s eta 0:00:00
Requirement already satisfied: wheel in /usr/local/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.4.0->accelerate) (0.38.4)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.4.0->accelerate) (63.2.0)
Collecting jinja2
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 399.1 MB/s eta 0:00:00
Collecting pydub
  Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)
Requirement already satisfied: fastapi in /usr/local/lib/python3.10/site-packages (from gradio~=3.6) (0.75.2)
Collecting pandas
  Downloading pandas-1.5.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.1/12.1 MB 467.5 MB/s eta 0:00:00
Collecting ffmpy
  Downloading ffmpy-0.3.0.tar.gz (4.8 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting markdown-it-py[linkify,plugins]
  Downloading markdown_it_py-2.1.0-py3-none-any.whl (84 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.5/84.5 kB 418.3 MB/s eta 0:00:00
Collecting httpx
  Downloading httpx-0.23.1-py3-none-any.whl (84 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.0/85.0 kB 428.2 MB/s eta 0:00:00
Collecting paramiko
  Downloading paramiko-2.12.0-py2.py3-none-any.whl (213 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 213.1/213.1 kB 469.7 MB/s eta 0:00:00
Requirement already satisfied: pydantic in /usr/local/lib/python3.10/site-packages (from gradio~=3.6) (1.10.2)
Collecting fsspec
  Downloading fsspec-2022.11.0-py3-none-any.whl (139 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.5/139.5 kB 443.0 MB/s eta 0:00:00
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/site-packages (from gradio~=3.6) (3.8.1)
Collecting h11<0.13,>=0.11
  Downloading h11-0.12.0-py3-none-any.whl (54 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.9/54.9 kB 374.7 MB/s eta 0:00:00
Collecting orjson
  Downloading orjson-3.8.2-cp310-cp310-manylinux_2_28_x86_64.whl (144 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 144.8/144.8 kB 411.8 MB/s eta 0:00:00
Collecting pycryptodome
  Downloading pycryptodome-3.15.0-cp35-abi3-manylinux2010_x86_64.whl (2.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 466.4 MB/s eta 0:00:00
Requirement already satisfied: python-multipart in /usr/local/lib/python3.10/site-packages (from gradio~=3.6) (0.0.5)
Collecting matplotlib
  Downloading matplotlib-3.6.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.8/11.8 MB 477.6 MB/s eta 0:00:00
Collecting uvicorn
  Downloading uvicorn-0.20.0-py3-none-any.whl (56 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.9/56.9 kB 389.7 MB/s eta 0:00:00
Collecting websockets>=10.0
  Downloading websockets-10.4-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (106 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 106.8/106.8 kB 357.9 MB/s eta 0:00:00
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1
  Downloading tokenizers-0.13.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/7.6 MB 211.0 MB/s eta 0:00:00
Collecting tqdm>=4.27
  Downloading tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.5/78.5 kB 377.2 MB/s eta 0:00:00
Requirement already satisfied: wcwidth>=0.2.5 in /usr/local/lib/python3.10/site-packages (from ftfy) (0.2.5)
Collecting absl-py>=0.4
  Downloading absl_py-1.3.0-py3-none-any.whl (124 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.6/124.6 kB 419.3 MB/s eta 0:00:00
Collecting werkzeug>=1.0.1
  Downloading Werkzeug-2.2.2-py3-none-any.whl (232 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 232.7/232.7 kB 448.8 MB/s eta 0:00:00
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Downloading google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting google-auth<3,>=1.6.3
  Downloading google_auth-2.14.1-py2.py3-none-any.whl (175 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.4/175.4 kB 430.4 MB/s eta 0:00:00
Collecting protobuf<4,>=3.9.2
  Downloading protobuf-3.20.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 468.7 MB/s eta 0:00:00
Collecting tensorboard-data-server<0.7.0,>=0.6.0
  Downloading tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.9/4.9 MB 484.5 MB/s eta 0:00:00
Collecting tensorboard-plugin-wit>=1.6.0
  Downloading tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 781.3/781.3 kB 498.6 MB/s eta 0:00:00
Collecting grpcio>=1.24.3
  Downloading grpcio-1.51.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/4.8 MB 462.2 MB/s eta 0:00:00
Collecting markdown>=2.6.8
  Downloading Markdown-3.4.1-py3-none-any.whl (93 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 93.3/93.3 kB 392.4 MB/s eta 0:00:00
Requirement already satisfied: charset-normalizer<3,>=2 in /usr/local/lib/python3.10/site-packages (from requests) (2.1.1)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/site-packages (from requests) (2021.10.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/site-packages (from requests) (1.26.12)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/site-packages (from requests) (3.4)
Collecting kdtree
  Downloading kdtree-0.16-py2.py3-none-any.whl (7.7 kB)
Collecting pathtools
  Downloading pathtools-0.1.2.tar.gz (11 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: sentry-sdk>=1.0.0 in /usr/local/lib/python3.10/site-packages (from wandb) (1.5.1)
Collecting promise<3,>=2.0
  Downloading promise-2.3.tar.gz (19 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting docker-pycreds>=0.4.0
  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)
Requirement already satisfied: six>=1.13.0 in /usr/local/lib/python3.10/site-packages (from wandb) (1.16.0)
Collecting GitPython>=1.0.0
  Downloading GitPython-3.1.29-py3-none-any.whl (182 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 182.5/182.5 kB 421.9 MB/s eta 0:00:00
Collecting setproctitle
  Downloading setproctitle-1.3.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)
Requirement already satisfied: Click!=8.0.0,>=7.0 in /usr/local/lib/python3.10/site-packages (from wandb) (8.1.3)
Collecting shortuuid>=0.5.0
  Downloading shortuuid-1.0.11-py3-none-any.whl (10 kB)
Collecting gitdb<5,>=4.0.1
  Downloading gitdb-4.0.9-py3-none-any.whl (63 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.1/63.1 kB 386.2 MB/s eta 0:00:00
Collecting rsa<5,>=3.1.4
  Downloading rsa-4.9-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 436.0 MB/s eta 0:00:00
Collecting cachetools<6.0,>=2.0.0
  Downloading cachetools-5.2.0-py3-none-any.whl (9.3 kB)
Collecting requests-oauthlib>=0.7.0
  Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.10/site-packages (from packaging>=20.0->accelerate) (3.0.9)
Collecting MarkupSafe>=2.1.1
  Downloading MarkupSafe-2.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/site-packages (from aiohttp->gradio~=3.6) (6.0.2)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/site-packages (from aiohttp->gradio~=3.6) (4.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/site-packages (from aiohttp->gradio~=3.6) (1.8.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/site-packages (from aiohttp->gradio~=3.6) (22.1.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/site-packages (from aiohttp->gradio~=3.6) (1.3.3)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/site-packages (from aiohttp->gradio~=3.6) (1.3.1)
Requirement already satisfied: starlette==0.17.1 in /usr/local/lib/python3.10/site-packages (from fastapi->gradio~=3.6) (0.17.1)
Requirement already satisfied: anyio<4,>=3.0.0 in /usr/local/lib/python3.10/site-packages (from starlette==0.17.1->fastapi->gradio~=3.6) (3.6.2)
Requirement already satisfied: sniffio in /usr/local/lib/python3.10/site-packages (from httpx->gradio~=3.6) (1.3.0)
Collecting rfc3986[idna2008]<2,>=1.3
  Downloading rfc3986-1.5.0-py2.py3-none-any.whl (31 kB)
Collecting httpcore<0.17.0,>=0.15.0
  Downloading httpcore-0.16.1-py3-none-any.whl (68 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.7/68.7 kB 407.1 MB/s eta 0:00:00
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/site-packages (from importlib-metadata->diffusers>==0.5.0) (3.10.0)
Collecting mdurl~=0.1
  Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Collecting mdit-py-plugins
  Downloading mdit_py_plugins-0.3.1-py3-none-any.whl (46 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.5/46.5 kB 349.4 MB/s eta 0:00:00
Collecting linkify-it-py~=1.0
  Downloading linkify_it_py-1.0.3-py3-none-any.whl (19 kB)
Collecting fonttools>=4.22.0
  Downloading fonttools-4.38.0-py3-none-any.whl (965 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 965.4/965.4 kB 508.1 MB/s eta 0:00:00
Collecting cycler>=0.10
  Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.4.4-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 488.5 MB/s eta 0:00:00
Collecting contourpy>=1.0.1
  Downloading contourpy-1.0.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (296 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 296.1/296.1 kB 474.2 MB/s eta 0:00:00
Collecting python-dateutil>=2.7
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 497.3 MB/s eta 0:00:00
Collecting pytz>=2020.1
  Downloading pytz-2022.6-py2.py3-none-any.whl (498 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 498.1/498.1 kB 505.2 MB/s eta 0:00:00
Collecting cryptography>=2.5
  Downloading cryptography-38.0.3-cp36-abi3-manylinux_2_28_x86_64.whl (4.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 352.8 MB/s eta 0:00:00
Collecting bcrypt>=3.1.3
  Downloading bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl (593 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 593.7/593.7 kB 512.2 MB/s eta 0:00:00
Collecting pynacl>=1.0.1
  Downloading PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (856 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 856.7/856.7 kB 492.1 MB/s eta 0:00:00
Collecting cffi>=1.12
  Downloading cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 441.8/441.8 kB 480.4 MB/s eta 0:00:00
Collecting smmap<6,>=3.0.1
  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)
Collecting httpcore<0.17.0,>=0.15.0
  Downloading httpcore-0.16.0-py3-none-any.whl (68 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.7/68.7 kB 399.9 MB/s eta 0:00:00
  Downloading httpcore-0.15.0-py3-none-any.whl (68 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.4/68.4 kB 392.6 MB/s eta 0:00:00
Collecting uc-micro-py
  Downloading uc_micro_py-1.0.1-py3-none-any.whl (6.2 kB)
Collecting pyasn1<0.5.0,>=0.4.6
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.1/77.1 kB 395.1 MB/s eta 0:00:00
Collecting oauthlib>=3.0.0
  Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.7/151.7 kB 472.6 MB/s eta 0:00:00
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 435.5 MB/s eta 0:00:00
Building wheels for collected packages: promise, ffmpy, pathtools
  Building wheel for promise (setup.py): started
  Building wheel for promise (setup.py): finished with status 'done'
  Created wheel for promise: filename=promise-2.3-py3-none-any.whl size=21486 sha256=e3187f85d1b8b4195d3249a8486b96136d95dcc020457397e227cd7586033e95
  Stored in directory: /tmp/pip-ephem-wheel-cache-vf91vl0p/wheels/76/40/54/417a4d64a01b61b247658d83597e1dc83c3de01fc0cef44972
  Building wheel for ffmpy (setup.py): started
  Building wheel for ffmpy (setup.py): finished with status 'done'
  Created wheel for ffmpy: filename=ffmpy-0.3.0-py3-none-any.whl size=4693 sha256=87917d0dbb2be4d2928a7878bd6c8ba60f8d9137b45003bf78228d48da4a4511
  Stored in directory: /tmp/pip-ephem-wheel-cache-vf91vl0p/wheels/fe/17/e9/577da024bc5aede641c69f0675254c1e518db79800abbe135c
  Building wheel for pathtools (setup.py): started
  Building wheel for pathtools (setup.py): finished with status 'done'
  Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8791 sha256=3944724db950a0a8d4096c0c4350eb581e34165a2ed9d2642c2b3b99e7734455
  Stored in directory: /tmp/pip-ephem-wheel-cache-vf91vl0p/wheels/44/1b/54/249c94316d4e1030e2d0683fba1d8ea06197de866f5a4de738
Successfully built promise ffmpy pathtools
Installing collected packages: tokenizers, tensorboard-plugin-wit, rfc3986, pytz, pydub, pyasn1, pathtools, kdtree, ffmpy, websockets, uc-micro-py, tqdm, tensorboard-data-server, smmap, smart_open, shortuuid, setproctitle, rsa, requests, regex, pyyaml, python-dotenv, python-dateutil, pycryptodome, pycparser, pyasn1-modules, psutil, protobuf, promise, Pillow, orjson, oauthlib, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cublas-cu11, mdurl, MarkupSafe, markdown, kiwisolver, h11, grpcio, ftfy, fsspec, fonttools, filelock, docker-pycreds, cycler, contourpy, cachetools, bcrypt, absl-py, werkzeug, uvicorn, requests-oauthlib, pandas, nvidia-cudnn-cu11, matplotlib, markdown-it-py, linkify-it-py, jinja2, huggingface-hub, httpcore, google-auth, gitdb, climage, cffi, transformers, torch, pynacl, modelcards, mdit-py-plugins, httpx, google-auth-oauthlib, GitPython, diffusers, cryptography, wandb, torchvision, tensorboard, paramiko, accelerate, gradio
  Attempting uninstall: protobuf
    Found existing installation: protobuf 4.21.9
    Uninstalling protobuf-4.21.9:
      Successfully uninstalled protobuf-4.21.9
Successfully installed GitPython-3.1.29 MarkupSafe-2.1.1 Pillow-9.3.0 absl-py-1.3.0 accelerate-0.14.0 bcrypt-4.0.1 cachetools-5.2.0 cffi-1.15.1 climage-0.1.3 contourpy-1.0.6 cryptography-38.0.3 cycler-0.11.0 diffusers-0.7.2 docker-pycreds-0.4.0 ffmpy-0.3.0 filelock-3.8.0 fonttools-4.38.0 fsspec-2022.11.0 ftfy-6.1.1 gitdb-4.0.9 google-auth-2.14.1 google-auth-oauthlib-0.4.6 gradio-3.10.1 grpcio-1.51.0 h11-0.12.0 httpcore-0.15.0 httpx-0.23.1 huggingface-hub-0.11.0 jinja2-3.1.2 kdtree-0.16 kiwisolver-1.4.4 linkify-it-py-1.0.3 markdown-3.4.1 markdown-it-py-2.1.0 matplotlib-3.6.2 mdit-py-plugins-0.3.1 mdurl-0.1.2 modelcards-0.1.6 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 oauthlib-3.2.2 orjson-3.8.2 pandas-1.5.1 paramiko-2.12.0 pathtools-0.1.2 promise-2.3 protobuf-3.20.3 psutil-5.9.4 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pycryptodome-3.15.0 pydub-0.25.1 pynacl-1.5.0 python-dateutil-2.8.2 python-dotenv-0.21.0 pytz-2022.6 pyyaml-6.0 regex-2022.10.31 requests-2.28.1 requests-oauthlib-1.3.1 rfc3986-1.5.0 rsa-4.9 setproctitle-1.3.2 shortuuid-1.0.11 smart_open-6.2.0 smmap-5.0.0 tensorboard-2.11.0 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tokenizers-0.13.2 torch-1.13.0 torchvision-0.14.0 tqdm-4.64.1 transformers-4.24.0 uc-micro-py-1.0.1 uvicorn-0.20.0 wandb-0.13.5 websockets-10.4 werkzeug-2.2.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Creating image snapshot...
Finished snapshot; took 20.599490297s
Finished image build for im-buzjgafjcddrzhojrbxaxudrxlnh; took 78.7414474s
Building image im-ggbgbvfjevtvcbzgyicplgqkffzn

=> Step 0: FROM base

=> Step 1: RUN apt-get update
Hit:1 http://deb.debian.org/debian bullseye InRelease
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [204 kB]
Fetched 296 kB in 1s (404 kB/s)
Reading package lists...

=> Step 2: RUN apt-get install -y wget
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libpsl5 publicsuffix
The following NEW packages will be installed:
  libpsl5 publicsuffix wget
0 upgraded, 3 newly installed, 0 to remove and 4 not upgraded.
Need to get 1149 kB of archives.
After this operation, 4001 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main amd64 libpsl5 amd64 0.21.0-1.2 [57.3 kB]
Get:2 http://deb.debian.org/debian bullseye/main amd64 wget amd64 1.21-1+deb11u1 [964 kB]
Get:3 http://deb.debian.org/debian bullseye/main amd64 publicsuffix all 20220811.1734-0+deb11u1 [127 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 1149 kB in 0s (37.2 MB/s)
Selecting previously unselected package libpsl5:amd64.
(Reading database ... 16247 files and directories currently installed.)
Preparing to unpack .../libpsl5_0.21.0-1.2_amd64.deb ...
Unpacking libpsl5:amd64 (0.21.0-1.2) ...
Selecting previously unselected package wget.
Preparing to unpack .../wget_1.21-1+deb11u1_amd64.deb ...
Unpacking wget (1.21-1+deb11u1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../publicsuffix_20220811.1734-0+deb11u1_all.deb ...
Unpacking publicsuffix (20220811.1734-0+deb11u1) ...
Setting up libpsl5:amd64 (0.21.0-1.2) ...
Setting up wget (1.21-1+deb11u1) ...
Setting up publicsuffix (20220811.1734-0+deb11u1) ...
Processing triggers for libc-bin (2.31-13+deb11u5) ...
Creating image snapshot...
Finished snapshot; took 541.062646ms
Finished image build for im-ggbgbvfjevtvcbzgyicplgqkffzn; took 4.335535875s

Token will not been saved to git credential helper. Pass `add_to_git_credential=True` if you want to set the git credential as well.
Token is valid.
Your token has been saved to /root/.huggingface/token
Login successful
--2022-11-22 05:01:19--  https://raw.githubusercontent.com/huggingface/diffusers/30220905c4319e46e114cf7dc8047d94eca226f7/examples/dreambooth/train_dreambooth.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26448 (26K) [text/plain]
Saving to: ‘train_dreambooth.py’

train_dreambooth.py                           100%[================================================================================================>]  25.83K  --.-KB/s    in 0.002s

2022-11-22 05:01:19 (14.1 MB/s) - ‘train_dreambooth.py’ saved [26448/26448]

Traceback (most recent call last):████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.06M/1.06M [00:00<00:00, 3.18MB/s]
                                    File "/usr/local/bin/grcat", line 306, in <module>
                                                                                          myprint(nline)
                                                                                                        BlockingIOError: [Errno 35] write could not complete without blocking
charlesfrye commented 1 year ago

thanks for reporting!

that's an error that Modal raises when too much information is written to the stdout at once.

it seems to be dependent on nondeterministic state in the concurrent executions.

in general, i've found that just re-running a failed command resolves the error

Klaudioz commented 1 year ago

I tried again, and I got the same, but this time faster:

image
erikbern commented 1 year ago

Interesting – we'll take a look at it!

aksh-at commented 1 year ago

Will look into it tomorrow, but it might have to do with the write buffer getting full while in interactive mode. Just noticed its enabled by default: https://github.com/charlesfrye/dreambooth-modal/blob/aba8dc8da739e15b980bbf02595262169678bd8f/run.py#L93

Probably makes sense to turn it on while debugging (when you need to do something interactive like run IPython), but it's a good idea to disable it otherwise.

Klaudioz commented 1 year ago

Actually, the terminal got so blocked, that Ctrl-C or Ctrl-D stopped to work, and I had to close it.

Thanks for your help!!.

Klaudioz commented 1 year ago

I tried it again today and I got the same error 😢