how to fine tune the bge model by using single GPU?? how to set or how to edit code

[Yazooliu]

After prepare the training env , I try to finetune the model as following the step2 and step3 in https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune#hard-negatives

step2 is done and can generated the xxxHN.jsonl by following step3, it will raise the CUDA of memorry. My GPU1 memory is full but GPU2 can ok. How I can to edit the code or change to us one GPU to finetune.

My GPU info as following:

and trianing command as:

or do you hava any good suggetion to us less GPU resource or set to GPU No.2 to run it successfully? I only have 70 query with positive and negative example to finetune as the data is too less.

Thanks for help.

[staoxiao]

You can set export CUDA_VISIBLE_DEVICES=1 to use GPU 1. The detail can refer to https://stackoverflow.com/questions/39649102/how-do-i-select-which-gpu-to-run-a-job-on .

[Yazooliu]

You can set export CUDA_VISIBLE_DEVICES=1 to use GPU 1. The detail can refer to https://stackoverflow.com/questions/39649102/how-do-i-select-which-gpu-to-run-a-job-on .

other issue occur:

[staoxiao]

delete --fp16

[Yazooliu]

delete --fp16

yes. it works. but maybe some issue in code as following????

this is code:

this is logger output info and error detail, (1) logger output info

---

09/26/2023 16:47:07 - WARNING - main - Process rank: 0, device: cpu, n_gpu: 0, distributed training: True, 16-bits training: False 09/26/2023 16:47:07 - INFO - main - Training/evaluation parameters RetrieverTrainingArguments( _n_gpu=0, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=False, bf16_full_eval=False, data_seed=None, dataloader_drop_last=True, dataloader_num_workers=0, dataloader_pin_memory=True, ddp_backend=None, ddp_broadcast_buffers=None, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=1800, debug=[], deepspeed=None, disable_tqdm=False, dispatch_batches=None, do_eval=False, do_predict=False, do_train=False, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fix_position_embedding=False, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, gradient_accumulation_steps=1, gradient_checkpointing=False, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_always_push=False, hub_model_id=None, hub_private_repo=False, hub_strategy=every_save, hub_token=, ignore_data_skip=False, include_inputs_for_metrics=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=1e-05, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=passive, log_level_replica=warning, log_on_each_node=True, logging_dir=BAAI/fine_tune_bge-large-en/runs/Sep26_16-47-07_gpu-server-worker-1, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=10, logging_strategy=steps, lr_scheduler_type=linear, max_grad_norm=1.0, max_steps=-1, metric_for_best_model=None, mp_parameters=, negatives_cross_device=True, no_cuda=False, num_train_epochs=5.0, optim=adamw_torch, optim_args=None, output_dir=BAAI/fine_tune_bge-large-en, overwrite_output_dir=False, past_index=-1, per_device_eval_batch_size=8, per_device_train_batch_size=32, prediction_loss_only=False, push_to_hub=False, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, ray_scope=last, remove_unused_columns=True, report_to=[], resume_from_checkpoint=None, run_name=BAAI/fine_tune_bge-large-en, save_on_each_node=False, save_safetensors=False, save_steps=500, save_strategy=steps, save_total_limit=None, seed=42, sharded_ddp=[], skip_memory_metrics=True, temperature=0.02, tf32=None, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torchdynamo=None, tpu_metrics_debug=False, tpu_num_cores=None, use_cpu=False, use_ipex=False, use_legacy_prediction_loop=False, use_mps_device=False, warmup_ratio=0.0, warmup_steps=0, weight_decay=0.0, ) 09/26/2023 16:47:07 - INFO - main - Model parameters ModelArguments(model_name_or_path='BAAI/bge-large-en', config_name=None, tokenizer_name=None, cache_dir=None, sentence_pooling_method='cls', normlized=True) 09/26/2023 16:47:07 - INFO - main - Data parameters DataArguments(train_data='./bge_fine_tune_data_example.jsonl', train_group_size=5, query_max_len=64, passage_max_len=500, max_example_num_per_dataset=100000000, query_instruction_for_retrieval='为这个句子生成表示以用于检索相关文章：', passage_instruction_for_retrieval=None) 09/26/2023 16:47:07 - INFO - main - Config: BertConfig { "_name_or_path": "BAAI/bge-large-en",

error info:

It seems that 'RetrieverTrainingArguments' object has no attribute 'sentence_pooling_method'

[staoxiao]

You can use the latest code. git clone the repo or pip install -U FlagEmbedding

[Yazooliu]

You can use the latest code. git clone the repo or pip install -U FlagEmbedding

seems your code support 2 diff generated embedding method. which will be better?

[staoxiao]

For bge, cls maybe better because we use cls in training. However, you can select different settings based on the performance of your data.

[Yazooliu]

For bge, cls maybe better because we use cls in training. However, you can select different settings based on the performance of your data.

If I hava any good change or idea, could I support to PR? Any limitaiton from your side?

[Yazooliu]

You can use the latest code. git clone the repo or pip install -U FlagEmbedding your this pip install method can not support this:

[staoxiao]

For bge, cls maybe better because we use cls in training. However, you can select different settings based on the performance of your data.

If I hava any good change or idea, could I support to PR? Any limitaiton from your side?

Sure. PR is very welcome, and there is no limitation.

[staoxiao]

You can use the latest code. git clone the repo or pip install -U FlagEmbedding your this pip install method can not support this:

Sorry, I forgot to add the sentence_transformers to setup.py. You can install sentence_transformers manually.