Questions about experimental code

[yongrenr]

Hello, I'm very interested in your model, about the genome benchmark, I operated through your guidance, and there were 2 problems - the dataloader length is 0 and the loss is infinite, I don't know if this is normal, can you help confirm what the reason is? Q1: RUN: python -m train \ experiment=hg38/genomic_benchmark \ callbacks.model_checkpoint_every_n_steps.every_n_train_steps=5000 \ dataset.dataset_name="dummy_mouse_enhancers_ensembl" \ dataset.train_val_split_seed=1 \ dataset.batch_size=128 \ dataset.rc_aug=false \ +dataset.conjoin_train=false \ +dataset.conjoin_test=false \ loader.num_workers=2 \ model=caduceus \ model.name=dna_embedding_caduceus \ +model.config_path="" \ +model.conjoin_test=false \ +decoder.conjoin_train=true \ +decoder.conjoin_test=false \ optimizer.lr="1e-3" \ trainer.max_epochs=10 \ train.pretrained_model_path="<path to .ckpt file>" \ wandb=null ERROR:

Q2: RUN: python -m train \ experiment=hg38/hg38 \ callbacks.model_checkpoint_every_n_steps.every_n_train_steps=500 \ dataset.max_length=1024 \ dataset.batch_size=1024 \ dataset.mlm=true \ dataset.mlm_probability=0.15 \ dataset.rc_aug=false \ model=caduceus \ model.config.d_model=128 \ model.config.n_layer=4 \ model.config.bidirectional=true \ model.config.bidirectional_strategy=add \ model.config.bidirectional_weight_tie=true \ model.config.rcps=true \ optimizer.lr="8e-3" \ train.global_batch_size=8 \ trainer.max_steps=10000 \ +trainer.val_check_interval=10000 \ wandb=null ERROR:

[yair-schiff]

Regarding Q1, this is an error I haven't hit before. Can you provide a bit more of the console output. Also it looks like these two fields are empty in the command you used to launch. They need to be filled with arguments that correspond to a pre-trained model.

+model.config_path=""
train.pretrained_model_path="<path to .ckpt file>" `

---

Regarding Q2, can you post the LR and training loss graphs from wandb? Did the model ever hit a nan loss during training?

[yongrenr]

Q1:Sorry, it's my fault. The code I uploaded has issues, and here are more error screenshots. RUN： python -m train \ experiment=hg38/genomic_benchmark \ callbacks.model_checkpoint_every_n_steps.every_n_train_steps=5000 \ dataset.dataset_name="human_nontata_promoters" \ dataset.train_val_split_seed=2 \ dataset.batch_size=128 \ dataset.rc_aug=false \ +dataset.conjoin_train=false \ +dataset.conjoin_test=false \ loader.num_workers=2 \ model=caduceus \ model.name=dna_embedding_caduceus \ +model.config_path="/home/gyc/caduceus-main/outputs/2024-03-11/20-21-19-995417/model_config.json" \ +model.conjoin_test=false \ +decoder.conjoin_train=true \ +decoder.conjoin_test=false \ optimizer.lr="1e-3" \ trainer.max_epochs=10 \ train.pretrained_model_path="/home/gyc/caduceus-main/outputs/2024-03-11/20-21-19-995417/checkpoints/last.ckpt" \ wandb=null ERROR：

[yair-schiff]

I just tried running this and did not hit the division by zero error. Can you confirm that data was properly downloaded to ./data/genomic_benchmark/human_nontata_promoters/ by the genomics-benchmark library:

This directory should look like this

data/genomic_benchmark/human_nontata_promoters/
├── test
│   ├── negative
│   └── positive
└── train
    ├── negative
    └── positive

these directories should contain .txt files with sequences.

[yongrenr]

Thanks for the reminder, I've successfully run your code and it works great!

[yair-schiff]

Glad to hear it!