MCC value problem on Nucleotide Transformer

[WeiminWu2000]

I really appreciate your work, but when I train model on the downstream task Nucleotide Transformer, it shows that no MCC result returned, the details are as follows, could you please help me to solve it, thank you very mcuh! Traceback (most recent call last): File "/root/code/dnabert3/hyena-dna/train.py", line 692, in main train(config) File "/root/code/dnabert3/hyena-dna/train.py", line 673, in train trainer.fit(model) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit call._call_and_handle_interrupt( File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt return trainer_fn(*args, *kwargs) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 645, in _fit_impl self._run(model, ckpt_path=self.ckpt_path) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1098, in _run results = self._run_stage() File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1177, in _run_stage self._run_train() File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1200, in _run_train self.fit_loop.run() File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run self.on_advance_end() File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 295, in on_advance_end self.trainer._call_callback_hooks("on_train_epoch_end") File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1380, in _call_callback_hooks fn(self, self.lightning_module, args, **kwargs) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 312, in on_train_epoch_end self._save_topk_checkpoint(trainer, monitor_candidates) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 367, in _save_topk_checkpoint raise MisconfigurationException(m) lightning_lite.utilities.exceptions.MisconfigurationException: ModelCheckpoint(monitor='val/mcc') could not find the monitored key in the returned metrics: ['trainer/loss', 'trainer/epoch', 'val/accuracy', 'val/loss', 'train/accuracy', 'train/loss', 'epoch', 'step']. HINT: Did you call log('val/mcc', value) in the LightningModule?

[exnx]

1. Are you running the docker image? or default repo code?

2. It seems like a config changed and you are no longer passing the val/mcc metric to Lightning. You need this in your config somewhere, which is currently in the configs/pipeline/nucleotide_transformer.yaml.

train:
  monitor: val/${dataset.metric}
  mode: max

[exnx]

Not sure what else to tell you, if you follow the previous comment it will calculate the mcc.

If there’s some other custom functionality you need then you’ll need to get very familiar with Pytorch Lightning. Good luck!

On Fri, Nov 24, 2023 at 2:31 PM WeiminWu2000 @.***> wrote:

Thanks for your reply. I am running the default repo. Yes, I learn what you mean, but I want to calculate the MCC value and output it. I think maybe it does not calculate or log the MCC, could I modify something to output the MCC other than the accuracy (it works well when I use accuracy) Besides, I want to perform a train-val-test experiment on dataset Nucleotide Transformer. In the default setting, the code onlys performs train and test without validating. I use the code on line 675 in train.py, (i.e. trainer.test(model)), but it has the following problem: Traceback (most recent call last): File "/root/code/dnabert3/hyena-dna/train.py", line 690, in main train(config) File "/root/code/dnabert3/hyena-dna/train.py", line 675, in train trainer.test(model) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 780, in test return call._call_and_handle_interrupt( File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt return trainer_fn(*args, kwargs) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 829, in _test_impl results = self._run(model, ckpt_path=self.ckpt_path) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1098, in _run results = self._run_stage() File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1174, in _run_stage return self._run_evaluate() File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1214, in _run_evaluate eval_loop_results = self._evaluation_loop.run() File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run self.advance(*args, *kwargs) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 152, in advance dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run self.advance(args, kwargs) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 137, in advance output = self._evaluation_step(kwargs) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 234, in _evaluation_step output = self.trainer._call_strategy_hook(hook_name, kwargs.values()) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1480, in _call_strategy_hook output = fn(args, kwargs) File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 399, in test_step return self.model.test_step(*args, *kwargs) File "/root/code/dnabert3/hyena-dna/train.py", line 440, in test_step return self._shared_step( File "/root/code/dnabert3/hyena-dna/train.py", line 338, in _shared_step torchmetrics = getattr(self, f'{prefix}_torchmetrics') File "/root/miniconda3/envs/hyena-dna/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1265, in getattr* raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'SequenceLightningModule' object has no attribute 'final/val_torchmetrics'

Could you please help me to solve them, thank you very much!

— Reply to this email directly, view it on GitHub https://github.com/HazyResearch/hyena-dna/issues/31#issuecomment-1826116871, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGNEKPCDNXEIUGEVI6KLCTTYGEN4ZAVCNFSM6AAAAAA7ZMXDP2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGEYTMOBXGE . You are receiving this because you commented.Message ID: @.***>

[WeiminWu2000]

Thanks very much. I'll learn about it.

[exnx]

Actually, what I showed above grabs the default metric set by the Nucleotide Transformer benchmarks, so it could be acc, or mcc. To force it show mcc, you can hardcode it, like this.

Note, this is what Lightning uses to save the best checkpoints, it "monitors" the val/mcc and will save the "max" or highest mcc ckpts.

train:
  monitor: val/mcc
  mode: max

[WeiminWu2000]

Thanks for your reply! I'll try it!

[exnx]

looks like that worked

[WeiminWu2000]

Yes, thanks very much for you help!