search

gguibon / metalearning-emotion-datasource

In this paper, we place ourselves in a classification scenario in which the target classes and data type are not accessible during training. We use a meta-learning approach to determine whether or not meta-trained information from common social network data with fine-grained emotion labels can achieve competitive performance on messages labeled with different emotion categories. We leverage few-shot learning to match with the classification scenario and consider metric learning based meta-learning by setting up Prototypical Networks with a Transformer encoder, trained in an episodic fashion. This approach proves to be effective for capturing meta-information from a source emotional tag set to predict previously unseen emotional tags. Even though shifting the data type triggers an expected performance drop, our meta-learning approach achieves decent results when compared to the fully supervised one.
GNU Affero General Public License v3.0
3 stars 1 forks source link

support set and query set? #1

Closed gghhoosstt closed 2 years ago

gghhoosstt commented 2 years ago

Hello , I have a question that does every task need to split into a support set and a query set? So, each episode samples support data from the support set and samples query data from the query set?

gguibon commented 2 years ago

Hello ghhoosstt,

Thanks for the question. Here is a clarification.

For each task (i.e. label), we randomly create an episode from this task's data. Hence, each episode samples support set and query set from the same data pool, while ensuring not to pick one element twice.

Your idea would be interesting to try. But this is not what we did in this work.

I hope it answers your question.

gghhoosstt commented 2 years ago

Thank you for your efficiency！ Consider that you meta-test for 1000 episodes in your paper, I also wonder that each episode samples support set and query set from the same data pool is true for meta-test?

gguibon commented 2 years ago

Yes it is.

To do meta-learning, we have 3 data pools: train, val, test which are defined by their set of labels (i.e. tasks).

During training: an episode's support & query sets come from the train data pool (from goemotions) During validation: an episode's support & query sets come from the val data pool (from goemotions) During test: an episode's support & query sets come from the test data pool. Plus, the data source (goemotions or dailydialog) can be changed.

Feel free to ask other questions.

gghhoosstt commented 2 years ago

ok ， thank you for clear reply. But, I have a new question: considering that you meta-test for 1000 episodes, suppose a case that example A is sampled from episode #1 as a support data, later, example A is sampled from episode #100 as a query data, Because example A has ever seen by the model in episode #1, how to compute accuracy ? I may have misunderstood you, Please advise me, thank you!

gguibon commented 2 years ago

Accuracy (and other metrics) are computed for each episode. Reported results are the average scores of all metrics and their standard deviation across episodes. This is why we set multiple queries per way, the computed score can then be much more precise. If you look at the code, you will see we compute metrics directly from the model base class's methods.

gghhoosstt commented 2 years ago

so can I understand during meta-test, each episode is finetune the model got in meta-train from scratch rather than continued finetune from last episode？

gguibon commented 2 years ago

During meta-test, the model is not fine-tuned, it only uses the support set to compute the prototypes for the current episode. In the paper in Table 2, we report the effect of fine-tuning the meta trained model (it makes the performance drop a lot).

gghhoosstt commented 2 years ago

ok，I see. Thank you very much!