Fine-tuning on a Sequence Classification Task

[leannmlindsey]

I am trying to fine tune the new pre-trained model EVO, which uses the stripedhyena architecture on a sequence classification task. When trying to use AutoModelForSequenceClassification from hugging face I get this error (see ERROR MESSAGE #1). I then tried to set up a simple torch.nn layer for fine-tuning but I get a message that the forward() layer does not expect the "label" keyword in the dataset. (see ERROR MESSAGE #2).

Any advice would be appreciated. We are very excited to use this new model on our tasks. Thank you

ERROR MESSAGE #1 Traceback (most recent call last): File "/home/llindsey1/CHPC/evo_finetune.py", line 32, in seq_classification_model = AutoModelForSequenceClassification.from_config(config) File "/home/llindsey1/.conda/envs/EVO/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 437, in from_config raise ValueError( ValueError: Unrecognized configuration class <class 'transformers_modules.togethercomputer.evo-1-131k-base.8eb9480ea22de5f86eeebc1199a76b63b42d7170.configuration_hyena.StripedHyenaConfig'> for this kind of AutoModel: AutoModelForSequenceClassification. Model type should be one of AlbertConfig, BartConfig, BertConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BloomConfig, CamembertConfig, CanineConfig, LlamaConfig, ConvBertConfig, CTRLConfig, Data2VecTextConfig, DebertaConfig, DebertaV2Config, DistilBertConfig, ElectraConfig, ErnieConfig, ErnieMConfig, EsmConfig, FalconConfig, FlaubertConfig, FNetConfig, FunnelConfig, GemmaConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTJConfig, IBertConfig, LayoutLMConfig, LayoutLMv2Config, LayoutLMv3Config, LEDConfig, LiltConfig, LlamaConfig, LongformerConfig, LukeConfig, MarkupLMConfig, MBartConfig, MegaConfig, MegatronBertConfig, MistralConfig, MixtralConfig, MobileBertConfig, MPNetConfig, MptConfig, MraConfig, MT5Config, MvpConfig, NezhaConfig, NystromformerConfig, OpenLlamaConfig, OpenAIGPTConfig, OPTConfig, PerceiverConfig, PersimmonConfig, PhiConfig, PLBartConfig, QDQBertConfig, Qwen2Config, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, SqueezeBertConfig, StableLmConfig, T5Config, TapasConfig, TransfoXLConfig, UMT5Config, XLMConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig, YosoConfig.

ERROR MESSAGE #2 File "/home/llindsey1/CHPC/evo_finetune_phage.py", line 96, in trainer.train() File "/home/llindsey1/.conda/envs/EVO/lib/python3.9/site-packages/transformers/trainer.py", line 1624, in train return inner_training_loop( File "/home/llindsey1/.conda/envs/EVO/lib/python3.9/site-packages/transformers/trainer.py", line 1961, in _inner_training_loop tr_loss_step = self.training_step(model, inputs) File "/home/llindsey1/.conda/envs/EVO/lib/python3.9/site-packages/transformers/trainer.py", line 2902, in training_step loss = self.compute_loss(model, inputs) File "/home/llindsey1/.conda/envs/EVO/lib/python3.9/site-packages/transformers/trainer.py", line 2925, in compute_loss outputs = model(inputs) File "/home/llindsey1/.conda/envs/EVO/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/llindsey1/.conda/envs/EVO/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) File "/home/llindsey1/.conda/envs/EVO/lib/python3.9/site-packages/accelerate/utils/operations.py", line 817, in forward return model_forward(*args, kwargs) File "/home/llindsey1/.conda/envs/EVO/lib/python3.9/site-packages/accelerate/utils/operations.py", line 805, in call return convert_to_fp32(self.model_forward(*args, *kwargs)) File "/home/llindsey1/.conda/envs/EVO/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast return func(args, kwargs) TypeError: forward() got an unexpected keyword argument 'label' 0%| | 0/100 [00:00<?, ?it/s]

[leannmlindsey]

Was able to solve this by changing the keyword to "labels" instead of "label"