jy0205 / Pyramid-Flow

Code of Pyramidal Flow Matching for Efficient Video Generative Modeling
https://pyramid-flow.github.io/
MIT License
1.98k stars 176 forks source link

Running app.py gives the error "Can't load tokenizer" #42

Closed MakingMadness closed 1 week ago

MakingMadness commented 1 week ago

Here's the full error:

$ python app.py 
Model directory '/home/zzz/ai/software/Pyramid-Flow/pyramid_flow_model' already exists. Skipping download.
using half precision
Using temporal causal attention
We interp the position embedding of condition latents
Traceback (most recent call last):
  File "app.py", line 47, in <module>
    model = PyramidDiTForVideoGeneration(
  File "/home/zzz/ai/software/Pyramid-Flow/pyramid_dit/pyramid_dit_for_video_gen_pipeline.py", line 99, in __init__
    self.text_encoder = SD3TextEncoderWithMask(model_path, torch_dtype=torch_dtype)
  File "/home/zzz/ai/software/Pyramid-Flow/pyramid_dit/modeling_text_encoder.py", line 19, in __init__
    self.tokenizer = CLIPTokenizer.from_pretrained(os.path.join(model_path, 'tokenizer'))
  File "/home/zzz/ai/software/miniconda3/envs/pyramid/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2070, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for '/home/zzz/ai/software/Pyramid-Flow/pyramid_flow_model/tokenizer'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/home/zzz/ai/software/Pyramid-Flow/pyramid_flow_model/tokenizer' is the correct path to a directory containing all relevant files for a CLIPTokenizer tokenizer.

And the contents of the folder in question:

$ ls -l pyramid_flow_model/tokenizer
total 1560
-rw-rw-r-- 1 zzz zzz  524619 Oct 11 22:15 tokenizer_merges.txt
-rw-rw-r-- 1 zzz zzz     588 Oct 11 22:15 tokenizer_special_tokens_map.json
-rw-rw-r-- 1 zzz zzz     705 Oct 11 22:15 tokenizer_tokenizer_config.json
-rw-rw-r-- 1 zzz zzz 1059962 Oct 11 22:15 tokenizer_vocab.json
MakingMadness commented 1 week ago

The solution was to delete pyramid_flow_model and let app.py redownload it.