search

kuleshov-group / caduceus

Bi-Directional Equivariant Long-Range DNA Sequence Modeling
Apache License 2.0
137 stars 14 forks source link

error when using hf model in genomic_benchmark pretrain #40

Closed lyksj200 closed 1 month ago

lyksj200 commented 1 month ago

It seems that I can use the model provided in the hugging face such as kuleshov-group/caduceus-ph_seqlen-1k_d_model-118_n_layer-4_lr-8e-3 and code in caduceus-hf_finetune to fine-tune model on the genomic_benchmark task. I download model and then use cmd: python -m train experiment=hg38/genomic_benchmark callbacks.model_checkpoint_every_n_steps.every_n_train_steps=5000 dataset.dataset_name="dummy_mouse_enhancers_ensembl" dataset.train_val_split_seed=1 dataset.batch_size=128 dataset.rc_aug=false +dataset.conjoin_train=false +dataset.conjoin_test=false loader.num_workers=2 model=caduceus model.name=dna_embedding_caduceus +model.config_path=/cpfs01/projects-HDD/cfff-282dafecea22_HDD/liuxuyang/projects/gene/caduceus-hf_finetune/kuleshov-group/caduceus-ps_seqlen-1k_d_model-118_n_layer-4_lr-8e-3/config.json +model.conjoin_test=false +decoder.conjoin_train=true +decoder.conjoin_test=false optimizer.lr="1e-3" trainer.max_epochs=10 train.pretrained_model_path=/cpfs01/projects-HDD/cfff-282dafecea22_HDD/liuxuyang/projects/gene/caduceus-hf_finetune/kuleshov-group/caduceus-ps_seqlen-1k_d_model-118_n_layer-4_lr-8e-3/model.safetensors wandb=null but it seems I didn't get the right config image

yair-schiff commented 1 month ago

This parameter model.config_path refers to a model config that would have been generated if you ran the pre-training as well. To use an HF model for this downstream task please use this branch: hf_finetune and see the discussion here

lyksj200 commented 1 month ago

Thank you for your answering. But I still can't use HF model using hf_finetune, my command is: python -m train experiment=hg38/genomic_benchmark callbacks.model_checkpoint_every_n_steps.every_n_train_steps=5000 dataset.dataset_name="dummy_mouse_enhancers_ensembl" dataset.train_val_split_seed=1 dataset.batch_size=128 dataset.rc_aug=false +dataset.conjoin_train=false +dataset.conjoin_test=false loader.num_workers=2 model=caduceus model.name=dna_embedding_caduceus +model.conjoin_test=false +decoder.conjoin_train=true +decoder.conjoin_test=false optimizer.lr="1e-3" trainer.max_epochs=10 train.pretrained_model_path=/cpfs01/projects-HDD/cfff-282dafecea22_HDD/liuxuyang/projects/gene/caduceus-hf_finetune/kuleshov-group/caduceus-ps_seqlen-1k_d_model-118_n_layer-4_lr-8e-3/model.safetensors wandb=null

and the error is image

yair-schiff commented 1 month ago

I believe you are still using the command / params for loading a model you pre-trained locally. Please instead refer to the commands/args defined in these files:

Namely, you will need to use model.pretrained_model_name_or_path="${PRETRAINED_PATH}" instead of train.pretrained_model_path, where PRETRAINED_PATH refers to either a HF path on the hub or locally

lyksj200 commented 1 month ago

Thank you for your excellent work and outstanding answers.