cannot run hg38_hyena_seqlen_warmup_reload

HazyResearch / hyena-dna

Official implementation for HyenaDNA, a long-range genomic foundation model built with Hyena

https://arxiv.org/abs/2306.15794

Apache License 2.0

532 stars 74 forks source link

cannot run hg38_hyena_seqlen_warmup_reload #45

Open sustcsonglin opened 5 months ago

sustcsonglin commented 5 months ago

Hi,

Thanks for your great work! I am trying to do hg38/hg38_hyena_seqlen_warmup_reload.yaml experiment. Got the following error msg:

I had some initial search on this issue and found this. I set monitor: test/loss and it still doesn't work. But i have no problem running 'g38/hg38_hyena.yaml'.

Do you have any insights on this issue? Is this related to the sentence length warmup callback? because I can run 'g38/hg38_hyena.yaml' without this callback. i am using pytorch_lightning v1.8.6

sustcsonglin commented 5 months ago

After some inspections, I found the reason! link1 disables loading the test dataloader during training, but the monitor metric was not changed. Hope this configuration will be fixed

sustcsonglin commented 5 months ago

Another question: does this configuration file consider the trainer.devices for setting accumulate_grad_batches? I didn't find sth like dividing by trainer.devices here