Training data description

[XinnuoXu]

Hi Yutai, nice work!

I am planning to train and test the model on other datasets. Do you mind to share the description of the form of the json files? Or does this repo offer any tools to create the json files from some basic data structure, like plain text with its NLU labels?

Cheers : )

[AtmaHou]

Thanks!

Here are the data descriptions:

1. At the file level, taking Snips data as an example, xval_snips contains cross-evaluation splits of all 1-shots SLU of snips (leave one domain test each time), and xval_snips_shot_5 is for 5 shots.
2. Within xval_snips, snips_test/train/dev_n.json is the test/train/dev data of the n_th cross-evaluation split.
3. For each data file, you can refer to the formate instructions here, https://github.com/AtmaHou/MetaDialog#few-shotmeta-episode-style-data-example .

For constructing such data from raw NLU data:

• We offer a toolkit described here: https://github.com/AtmaHou/MetaDialog#few-shot-data-construction-tool

[XinnuoXu]

Hi Yutai, many thanks for your reply.

Just to double check my understanding, for each cross-validation, you have 5 domains for training, 1 for validation and 1 for testing. During the training process, you (1) train the model on the 2000 * 5 training examples over the 5 training domains all at once (let's call it pre-trained model here) (2) test the model 100 times using the 100 few-shot episodes. In the step (2), for each few-shot episode in your test domain, you "fine-tuned" your pre-trained model with the 1-shot (or 5-shot) data in the support set and test the fine-tuned model with the 20 queries you sampled, right?

Another question is that how you constructed the training set? As far as I understood, you sampled 100 few-shot episodes for each domain. When you constructed the training set, you just used the 20 queries and their "ground truth" labels in each few-shot episode, in each domain, right?

Cheers : )

[XinnuoXu]

Sorry to bother you again Yutai. I notice that the random seed is critical to the performance. How did you make your decision to try the random seed candidates listed in your code? Do you have any suggestions for me if I want to adapt your model on other dataset like MultiWoZ or SGD?

[AtmaHou]

Hi~ High uncertainty is a common problem in few-shot models. In our experiment, we randomly select 5-10 seeds and then average them.

If you just want higher performance, you can use the early stop technique. The bad seeds will show up in the first few rounds, and you can simply filter them by thresholds.

[AtmaHou]

Hi Yutai, many thanks for your reply.

Just to double check my understanding, for each cross-validation, you have 5 domains for training, 1 for validation and 1 for testing. During the training process, you (1) train the model on the 2000 * 5 training examples over the 5 training domains all at once (let's call it pre-trained model here) (2) test the model 100 times using the 100 few-shot episodes. In the step (2), for each few-shot episode in your test domain, you "fine-tuned" your pre-trained model with the 1-shot (or 5-shot) data in the support set and test the fine-tuned model with the 20 queries you sampled, right?

Another question is that how you constructed the training set? As far as I understood, you sampled 100 few-shot episodes for each domain. When you constructed the training set, you just used the 20 queries and their "ground truth" labels in each few-shot episode, in each domain, right?

Cheers : )

There seems to be no problem, except that our method does not require any finetune process in the testing domains.

[XinnuoXu]

Thanks!

Here are the data descriptions:

1. At the file level, taking Snips data as an example, xval_snips contains cross-evaluation splits of all 1-shots SLU of snips (leave one domain test each time), and xval_snips_shot_5 is for 5 shots.
2. Within xval_snips, snips_test/train/dev_n.json is the test/train/dev data of the n_th cross-evaluation split.
3. For each data file, you can refer to the formate instructions here, https://github.com/AtmaHou/MetaDialog#few-shotmeta-episode-style-data-example .

For constructing such data from raw NLU data:

• We offer a toolkit described here: https://github.com/AtmaHou/MetaDialog#few-shot-data-construction-tool

Hi Yutai, thanks a lot for your replies. They are very helpful.

I met this error "No such file or directory: 'D:/Data/Project/Data/MetaData/config/config0.json'" when I was trying to create few-shot data for snips. Do you mind to share an example of this config?

Cheers : )

[XinnuoXu]

Hi Yutai, I left some comments in the other repo but haven't got replies. The data generated by the tool are missing keys like tokenized_texts, word_piece_labels, word_piece_marks that are required by the training. Could you please point me to the codes/configs that are used for the generation of these keys?

Cheers!

[XinnuoXu]

Hi Yutai, I left some comments in the other repo but haven't got replies. The data generated by the tool are missing keys like tokenized_texts, word_piece_labels, word_piece_marks that are required by the training. Could you please point me to the codes/configs that are used for the generation of these keys?

Cheers!

It's been solved by training with the scripts in the MetaDialog repo.

[AtmaHou]

Hi Yutai, I left some comments in the other repo but haven't got replies. The data generated by the tool are missing keys like tokenized_texts, word_piece_labels, word_piece_marks that are required by the training. Could you please point me to the codes/configs that are used for the generation of these keys? Cheers!

It's been solved by training with the scripts in the MetaDialog repo.

😊

[niloofar17]

Sorry to bother you again Yutai. I notice that the random seed is critical to the performance. How did you make your decision to try the random seed candidates listed in your code? Do you have any suggestions for me if I want to adapt your model on other dataset like MultiWoZ or SGD?

Hi, I was going through your comments and noticed that you are using the same datasets like Atis, snips and SGD. I was wondering if you can share the scripts with me.

cheers, Niloofar

[AtmaHou]

Sorry to bother you again Yutai. I notice that the random seed is critical to the performance. How did you make your decision to try the random seed candidates listed in your code? Do you have any suggestions for me if I want to adapt your model on other dataset like MultiWoZ or SGD?

Hi, I was going through your comments and noticed that you are using the same datasets like Atis, snips and SGD. I was wondering if you can share the scripts with me.

cheers, Niloofar

Maybe this will help？ https://github.com/AtmaHou/MetaDialog