How to add new category?

[b0tm1nd]

What is the scenario for adding a new category ?

[b0tm1nd]

From what I understood, we need a new dataset in .jsonl with text and labels. Could you share datasets that this was trained on? Especially for not_news. By reading the telegram contest I see that for russian content they mostly used lenta.ru archive. But what about ukrainian?

[NyanNyanovich]

Here you go: https://github.com/NyanNyanovich/nyan/releases/download/can_annot/cat_markup.tar.gz I used Lenta and gpt-4 annotations, here is the script to query gpt-4: https://github.com/NyanNyanovich/nyan/blob/master/scripts/annotate_categories.py And the training script: https://github.com/NyanNyanovich/nyan/blob/master/scripts/train_clf.py

[b0tm1nd]

@NyanNyanovich Thanks, I have found train_clf.py already and tried to train it with a single category but then on send.sh classificator failed probably because of "not_news" missing..

I have taken a dataset for Ukrainian news website which tagged their news, grouped only related to corruption and gotten about 700 entries which I united with categories_train.jsonl.

And after training I've became getting much worse results: many from war/politics became triggering corruption now and resulting as "unknown". I have found out that in the added dataset the median text size is 1000+ characters when in yours about 450.

So I have a few questions about the hints for a dataset for the new category:

1. Does smaller article size improves accuracy?
2. Do multiple labels for the new category (like ["corruption", "war"] or ["corruption", "politics"]) will increase accuracy?
3. What was your strategy (or was it random?) in news selection for your training dataset:

Labels sorted by Count: politics: 1200 occurrences war: 1062 occurrences economy: 760 occurrences incident: 699 occurrences not_news: 451 occurrences entertainment: 426 occurrences tech: 418 occurrences sports: 324 occurrences science: 138 occurrences other: 37 occurrences

4. What are the other hints you might suggest?