environment setup error and runtime error -- solution

[Cuberick-Orion]

Hi,

Thank you (the authors) for this very impressive work.

Below are a few notes that I've made while setting up the environment, in case if anyone ran into similar issues with the setup.

I am using the official pytorch docker pytorch/pytorch:1.12.1-cuda11.3-cudnn8-devel, note it comes with python 3.7, some packages (e.g., matplotlib) will complain with version mismatch, just remove the version specified in the requirement file, as the author states that they do not necessarily have to be bound to the specific versions).

A complete Dockerfile and requirement file is attached at the bottom. Below are some changes that I've made.

• pydantic Just noting that pip will default to pydantic==2.5.3 if not specified and it will cause error when import accelerate. To resolve it please specify installing pydantic==1.9.0 (I haven't tested training yet but at least that will remedy the above issue with the import).

• mmcv builds forever See this.

• xformer error with RuntimeError: CUTLASS submodule not found. Did you forget to rungit submodule update --init --recursive? This will solve it pip install git+https://github.com/facebookresearch/xformers.git@v0.0.13#egg=xformers. I found that 0.0.13 will run into a runetime error: CUDA error: an illegal memory access was encountered, which traces to

  │ /home/xxxxxxx/gitRepo_xxxx/SeerVideoLDM/seer/models/attention.py:630 in                          │
  │ _memory_efficient_attention_xformers                                                             │
  │                                                                                                  │
  │   627 │   │   if not mask is None:                                                               │
  │   628 │   │   │   mask = xformers.ops.LowerTriangularMask()                                      │
  │   629 │   │   # breakpoint()                                                                     │
  │ ❱ 630 │   │   hidden_states = xformers.ops.memory_efficient_attention(query, key, value, attn_   │
  │   631 │   │   hidden_states = self.reshape_batch_dim_to_heads(hidden_states)                     │
  │   632 │   │   return hidden_states                                                               │
  │   633                                                           

  Instead xformers==0.0.12 works flawlessly.

• Also add triton==2.1.0 to the requirement file.

---

Reproduce with:

Dockerfile

FROM pytorch/pytorch:1.12.1-cuda11.3-cudnn8-devel
# FROM nvidia/cuda:11.3.1-devel-ubuntu20.04

RUN apt-get update && apt-get install -y ffmpeg libsm6 libxext6 libglib2.0-0 git ca-certificates && apt-get clean
RUN nvcc --version

# install mmcv
RUN pip install openmim && mim install mmcv-full==1.7.0

COPY requirements.txt /tmp/
RUN pip install -r /tmp/requirements.txt && pip install --upgrade google-cloud-storage

modified requirement.txt

pydantic==1.9.0
accelerate==0.15.0
boto3==1.26.93
botocore==1.29.94
chainer==7.8.1
clip==0.2.0
cupy-cuda111==8.1.0
deepspeed==0.7.7
diffusers==0.10.2
einops==0.6.0
ftfy==6.1.1
gdown==4.6.0
gifmaker==1.5
huggingface-hub==0.11.1
imageio==2.25.0
matplotlib
mmdet==2.25.3
multiprocess==0.70.14
numpy
omegaconf==2.3.0
open-clip-torch==2.16.0
opencv-python==4.7.0.68
pandas
Pillow
pyparsing==3.0.9
rotary-embedding-torch==0.1.5
scikit-learn
scipy
six
taming-transformers==0.0.1
tensorboard==2.11.0
timm==0.6.12
tokenizers==0.13.2
tqdm
transformers==4.25.1
triton==2.1.0
xformers=0.0.12