search

netease-youdao / BCEmbedding

Netease Youdao's open-source embedding and reranker models for RAG products.
Apache License 2.0
1.3k stars 85 forks source link

accelerate加速，微调bce模型报错：Could not find the transformer layer class to wrap in the model. #66

Closed HuXiLiFeng closed 1 month ago

HuXiLiFeng commented 1 month ago

accelerate采用fsdp加速训练时，报错如下：

Exception: Could not find the transformer layer class to wrap in the model.

accelerate配置如下： compute_environment: LOCAL_MACHINE distributed_type: FSDP downcast_bf16: 'no' fsdp_config: fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP fsdp_backward_prefetch_policy: NO_PREFETCH fsdp_offload_params: false fsdp_sharding_strategy: 3 fsdp_state_dict_type: FULL_STATE_DICT fsdp_transformer_layer_cls_to_wrap: BertLayer machine_rank: 0 main_training_function: main mixed_precision: 'no' num_machines: 1 num_processes: 1 rdzv_backend: static same_network: true tpu_env: [] tpu_use_cluster: false tpu_use_sudo: false use_cpu: false

麻烦帮忙看看整个问题

shenlei1020 commented 1 month ago

fsdp_transformer_layer_cls_to_wrap: XLMRobertaLayer