zhangfaen / finetune-Qwen2-VL

MIT License
191 stars 18 forks source link

loading model after fine tuning #17

Closed zarif98sjs closed 1 week ago

zarif98sjs commented 1 week ago

After fine tuning with the sample test data, I tried to load the model to test. I keep getting this error:

Traceback (most recent call last):
  File "finetune-Qwen2-VL/test_on_trained_model_by_us.py", line 7, in <module>
    model = Qwen2VLForConditionalGeneration.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "finetune-Qwen2-VL/qwen2venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 3994, in from_pretrained
    with safe_open(resolved_archive_file, framework="pt") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer

I tried using my own data, and the same issue was there. I think all the model components are not saved successfully.

When I fine-tune, it ended with "Killed", but it didn't throw any error.

2024-11-03 16:39:34-finetune.py:241-INFO >> Batch 96 of epoch 1/1, average training loss of previous 2 batches: 0.30156277120113373
2024-11-03 16:39:35-finetune.py:241-INFO >> Batch 98 of epoch 1/1, average training loss of previous 2 batches: 0.04462078958749771
2024-11-03 16:39:36-finetune.py:241-INFO >> Batch 100 of epoch 1/1, average training loss of previous 2 batches: 0.16311094723641872
Killed

And I only got these files:

image

Any idea what might cause issue in saving?