Text Classifier using longformer

[pchankh]

Can we request to add a short example of longformer for long text/review classification? Current triviaQA is good but more examples will encourage further use of longformer.

Thanks. Patrick

[armancohan]

Yes, we have plans for adding a text classification example in the near future.

[ibeltagy]

In the meantime, you can probably just follow one of the text classification examples from huggingface here

[hendrycks]

Yes, we have plans for adding a text classification example in the near future.

I'm looking forward to this!

[Jordy-VL]

To help others who cannot wait for the text classification example, I suggest following this example from huggingface: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/trainer/01_text_classification.ipynb or similarly this (rather up-to-date) community blogpost: https://mccormickml.com/2019/07/22/BERT-fine-tuning/#a1-saving--loading-fine-tuned-model

Would be nice if we could get an update on when they expect the example to be posted. :) Otherwise I will give it a shot myself and post a link. Thanks for the great work!

[ibeltagy]

I agree. Thanks, @Jordy-VL

[vr25]

Is it possible to provide the same text classification code on IMDB and Hyperpartisan News mentioned in the last section of the paper? Maybe a link to the Hyperpartisan Dataset too? I found several datasets but I am looking for the one used in this paper.

I am just trying to understand how local and global attentions are applied to the [CLS] tokens, unlike BERT.

Thanks a lot!

[ibeltagy]

The dataset is here: https://pan.webis.de/semeval19/semeval19-web/

I am just trying to understand how local and global attentions are applied to the [CLS] tokens, unlike BERT.

You can check LongformerForSequenceClassification. We automatically set global attention here. If you want more details, it should be in the implementation of the self-attention. Does that answer your question?

[vr25]

Thanks a lot for your quick response, @ibeltagy .

Although I haven't yet implemented it (so I can have more questions later), that answers my basic question on distinguishing between local and global attention.

[HugToDebug]

Has this example been added? I can't find it, so I don't believe so, but is there an update?

[ibeltagy]

no, but as @Jordy-VL suggested earlier, it should be straightforward to use the existing text classification tools and scripts that were developed for BERT and the huggingface code. You just need to replace your AutoModel.from_pretrained('bert-base-cased') with AutoModel.from_pretrained('allenai/longformer-base-4096')

[Jordy-VL]

no, but as @Jordy-VL suggested earlier, it should be straightforward to use the existing text classification tools and scripts that were developed for BERT and the huggingface code. You just need to replace your AutoModel.from_pretrained('bert-base-cased') with AutoModel.from_pretrained('allenai/longformer-base-4096')

To follow-up on this, I managed to load the model for text classification with custom num_labels with the following snippet:

from transformers import AutoTokenizer, AutoConfig, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained('allenai/longformer-base-4096')
config = AutoConfig.from_pretrained('allenai/longformer-base-4096',num_labels=num_labels)
model = AutoModelForSequenceClassification.from_pretrained('allenai/longformer-base-4096',config=config)

[ibeltagy]

Closing. Please feel free to reopen or create a new one if needed.

[Nick9214]

In the meantime, you can probably just follow one of the text classification examples from huggingface here

The link is not valid anymore. Could you please repost it or any other material most recent?

[karelin]

Text classification example seemigly moved to legacy: https://github.com/huggingface/transformers/tree/master/examples/legacy/text-classification