Error in _init_weights function in modeling_bertabs.py

[shivangsharma1]

Hey there!

I am replicating the code and this paper for one of my personal projects. I am facing one issue when running :

config = BertAbsConfig(max_pos=args.block_size)
model = BertAbs.from_pretrained(
    'remi/bertabs-finetuned-cnndm-extractive-abstractive-summarization',
    config=config
)

The error I am getting is :

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
/tmp/ipykernel_14334/239094548.py in <cell line: 4>()
      2 
      3 
----> 4 model = BertAbs.from_pretrained(
      5     './model',
      6     config=config

~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   2377                 offload_index,
   2378                 error_msgs,
-> 2379             ) = cls._load_pretrained_model(
   2380                 model,
   2381                 state_dict,

~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/modeling_utils.py in _load_pretrained_model(cls, model, state_dict, loaded_keys, resolved_archive_file, pretrained_model_name_or_path, ignore_mismatched_sizes, sharded_metadata, _fast_init, low_cpu_mem_usage, device_map, offload_folder, offload_state_dict, dtype, load_in_8bit)
   2523             )
   2524             for module in uninitialized_modules:
-> 2525                 model._init_weights(module)
   2526 
   2527         # Make sure we are able to load base models as well as derived models (with heads)

~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/modeling_utils.py in _init_weights(self, module)
   1103         Initialize the weights. This method should be overridden by derived class.
   1104         """
-> 1105         raise NotImplementedError(f"Make sure `_init_weights` is implemented for {self.__class__}")
   1106 
   1107     def tie_weights(self):

NotImplementedError: Make sure `_init_weights` is implemented for <class 'modeling_bertabs.BertAbs'>

I am using the following versions: pytorch : '1.10.0' transformers : '4.25.1'

I believe this is more of a version issue, as this is very old implementation. Which library version was this code written?

Please let me know if there is any fresh implementation of summarization using BERT.

[HenryDashwood]

The blogpost was published on 24th March 2020 so probably Transformers 2.5.1 (https://pypi.org/project/transformers/#history).

There probably are more recent implementations using BERT or more recent LLMs