chainsawriot
commented
6 months ago By default, grafzahl uses almost the same default as the underlying simpletransformers, i.e. no freezing and all layers might get finetuned.
If you really want to freeze some layers, it is possible to do that with simpletransformers; but unfortunately, not possible with grafzahl. If you want to customize the finetuning at the layer level, I think you would be better off using simpletransformers or even transformers.
Given the discussion about which layer keeping as a token's representation in a down-streaming analysis (Jawahar et al., 2019; Ethayarajh, 2019) when for example using a pre-trained bert model, I was wondering if you are considering to allow to the user the possibility to select one specific layer when fine-tuning a Transformer via grafzahl. At the moment, which layer is consider when for example I specify [model_name = "bert-base-uncased"]? Thanks for your great package!