Fine-tuning for classification

[zas97]

Hello,

do you plan on adding the code to fine-tune your model in a classification task (like RVL-CDIP)?

PS : Nice work, splitting the layout and the language models is a really good idea!

[jpWang]

Hi, in fact, the code of classification task is very similar to that of NER task, since you only need to label [CLS] token with the corresponding classification category and label other tokens with the "ignore" flag (for example, -100 for torch.nn.CrossEntropyLoss).

[zas97]

Yes it seems like an option.

However, in the run_funsd.py script would replacing the AutoModelForTokenClassification with AutoModelForSequenceClassification and adapting the dataset to the classification task work?

[jpWang]

No, it wouldn't work, since there is no LiLTRobertaLikeForSequenceClassification implemented in https://github.com/jpWang/LiLT/blob/main/LiLTfinetune/models/LiLTRobertaLike/modeling_LiLTRobertaLike.py for now. You can implement it as you wish.

Or, if you want to implement classification task based on run_funsd.py, you can focus on https://github.com/jpWang/LiLT/blob/main/examples/run_funsd.py#L180-L208, and give the classification category id for [CLS] token to label_ids, and give -100 for other tokens to label_ids.