search

kamalkraj / e5-mistral-7b-instruct

Finetune mistral-7b-instruct for sentence embeddings
Apache License 2.0
61 stars 13 forks source link

Exception when "--checkpointing_steps" is set #12

Open Hypothesis-Z opened 2 months ago

Hypothesis-Z commented 2 months ago

Source code in Accelerate lib shows that weights in hooks is empty if the training task is launched via Deepspeed.

https://github.com/huggingface/accelerate/blob/b8c85839531ded28efb77c32e0ad85af2062b27a/src/accelerate/accelerator.py#L2778-L2824

Threrfore, IndexError will be raised in save_model_hook.

https://github.com/kamalkraj/e5-mistral-7b-instruct/blob/99021919b3c82bc67a4a897e8e9f39efe3d72cdc/peft_lora_embedding_semantic_search.py#L158-L162

liujiqiang999 commented 4 weeks ago

Hi, Have you solved this problem?

Hypothesis-Z commented 3 weeks ago

@liujiqiang999 Do not register the hooks.

accelerator.register_save_state_pre_hook(save_model_hook)

    # accelerator.register_load_state_pre_hook(load_model_hook)