search

DRAGNLabs / 301r_retnet

2 stars 1 forks source link

LongNet integration broke eval_suite.py #73

Open DrewGalbraith opened 2 months ago

DrewGalbraith commented 2 months ago

Commit a9dc6c7 from #63 breaks eval_suite.py.

It introduces the following error:

ValueError: The config you are passing has a `model_type` attribute that is not consistent with the model type you passed (config has custom_transformer and you passed longnet. Fix one of those so they match!

The longnet reference in eval_suite.py overwrites the custom_transformer registration on the line before. This line should be removed for the transformers to run correctly.

DrewGalbraith commented 2 months ago

A valid concern is that the LongNetConfig object will have a class atrribute model_type that doesn't match the autoregistered set up in eval_suite.py. Likely, this is why the line in question was added. This can be corrected by making a copy of the DecoderConfig class with a different model_type class attribute, then using that in the autoregistration instead of DecoderConfig.

DrewGalbraith commented 2 months ago

@JacksonSearle, is there a LongNet in the group repo yet that I can test this on?

DrewGalbraith commented 2 months ago

Also, while we're at it, we should uncomment lines 7 and 20 in eval_main.py.

They lead to unexpected behavior (read: a silent failure) commented out as they are now.