search

kuleshov-group / caduceus

Bi-Directional Equivariant Long-Range DNA Sequence Modeling
Apache License 2.0
137 stars 14 forks source link

"omegaconf.errors.InterpolationKeyError: Interpolation key 'metric' not found" in NucelotideTransformer Benchmark #36

Closed yangzhao1230 closed 1 month ago

yangzhao1230 commented 1 month ago

Hi,

When I tried to reproduce NT benchmark's results, I met an error that seemed to be related to the train.metric not set in configuration.

image

I used the script run_nucleotide_transformer.sh you provided and only added some configs needed by this script. I did't change any configs in the default config files.

image

I know this might be a simple question, but I'm really not familiar with HYDRA configuration parameters, so I'd like to seek your help this time.

yangzhao1230 commented 1 month ago

Additionally, I would like to ask about the evaluation standards for the NT benchmark. Do you report the average performance on the validation set directly? If so, does that mean you didn't use the test set provided by NT? The NT paper doesn't seem to be very clear on this point—it mentions some evaluations on the test set, which I find confusing. Thank you!