Open julianpollmann opened 1 year ago
It seems like this is caused by the trainer freezing the model weights and not adding adapter modules other than invertibles to the EncoderDecoderMixin.
The trainer will not get any parameters with requires_grad=True for the optimizer over here.
I tried to add these some layers to the Mixin like done in the BARTMixin without success:
def _init_adapter_modules(self):
if not isinstance(self.encoder, ModelAdaptersMixin) or not isinstance(self.decoder, ModelAdaptersMixin):
return
# Relay all invertible adapter calls to encoder
self.invertible_adapters = self.encoder.base_model.invertible_adapters
self.add_invertible_adapter = self.encoder.base_model.add_invertible_adapter
self.get_invertible_adapter = self.encoder.base_model.get_invertible_adapter
self.enable_invertible_adapters = self.encoder.base_model.enable_invertible_adapters
self.invertible_adapters_forward = self.encoder.base_model.invertible_adapters_forward
# Decoder should use invertible adapters of encoder
self.decoder.base_model.invertible_adapters = self.encoder.base_model.invertible_adapters
self.decoder.base_model.add_invertible_adapter = lambda *args: None
self.decoder.base_model.get_invertible_adapter = self.encoder.base_model.get_invertible_adapter
# Added these lines from BARTMixin
# Also tried with self.encoder.base_model...
self.encoder.attention_adapters = AdapterLayer("mh_adapter", self.config)
self.encoder.output_adapters = AdapterLayer("output_adapter", self.config)
self.encoder.attention_adapters._init_adapter_modules()
self.encoder.output_adapters._init_adapter_modules()
self.decoder.cross_attention_adapters = AdapterLayer("cross_adapter", self.config)
self.decoder.cross_attention_adapters._init_adapter_modules()
Any ideas @calpt @hSterz
Thanks for reporting these issues and sorry for not getting back to you earlier. Unfortunately, our current encoder-decoder implementation is very hacky and has all sorts of issues currently. We'll try to look into this.
Environment info
adapter-transformers
version: 3.2.1Details
I'm trying to train a EncoderDecoder Adapter with BertGeneration using the Seq2SeqAdapterTrainer:
This results in a RuntimeError:
This issue is also present in transformers, where a change of the optimizer is discussed as a solution. Changing the optimizer in the Seq2SeqAdapterTrainer did not work:
Training with the same settings without adapter and Seq2SeqTrainer does work.