tokenizer.json is empty after stage3 training

CircleRadon / Osprey

[CVPR2024] The code for "Osprey: Pixel Understanding with Visual Instruction Tuning"

Apache License 2.0

775 stars 42 forks source link

tokenizer.json is empty after stage3 training #39

Closed Z-MU-Z closed 3 months ago

Z-MU-Z commented 3 months ago

Hello, I find that after stage3 training the tokenizer.json in checkpoint dir is empty, so when I run evaluation, I am confronted with below error.

 File "/home/zmz/miniconda3/envs/osprey/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 111, in __init__
    fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
Exception: EOF while parsing a value at line 1 column 0

Z-MU-Z commented 3 months ago

I also find that pytorch_model.bin.index.json is empty

Z-MU-Z commented 3 months ago

I'm very sorry to bother you. I find this issue seems to have been caused by a transmission error.