Check flair models for static embeddings - Githubissues

CLARIN-PL / embeddings

Embeddings: State-of-the-art Text Representations for Natural Language Processing tasks, an initial version of library focus on the Polish Language

https://clarin-pl.github.io/embeddings/

MIT License

36 stars 3 forks source link

Check flair models for static embeddings #151

Closed ktagowski closed 2 years ago

ktagowski commented 2 years ago

Test flair models for static embeddings. Compare our pipeline results with results in literature or other leaderboards

djaniak commented 2 years ago

The static embeddings work fine with the flair pipelines, achieving satisfactory results when evaluated on polemo2 dataset (text classification). The HPS pipeline was run for 100 iterations with the following outcome:

HPS results

![image](https://user-images.githubusercontent.com/26749468/148930912-dd8f61a2-9fe5-4ad8-b7dc-8b794d57033a.png)

These results were compared with the ones reported in two papers:

original publication that introduced the datasets: Multi-Level Sentiment Analysis of PolEmo 2.0: Extended Corpus of Multi-Domain Consumer Reviews
and publication with KLEJ benchmark: KLEJ: Comprehensive Benchmark for Polish Language Understanding

Results were obtained with hotels and medicine domains.

Name	Score	Metric	Pooling	Embeddings
our (flair)	~0.83	F1-macro	CNN	fastText KGR10
KLEJ	~0.83	Accuracy	LSTM	fastText
original paper	~0.75	F1-macro	biLSTM	fastText KGR10