about word2count

[GT-KIM]

Hello,

I read the original repository to get word2count.pkl file. I can get word2count.pkl after following "Download the data" section. However, that word2count.pkl is for Convai2 data, not empathetic dialogues data.

Next, I tried to follow the "(Alternatively) Making the data yourself" section with empathetic dialogues data. However, raw data (.csv format) of the empathetic dialogues cannot convert to the ParlAI format or word2count.pkl. I changed the source code, opt, task for migration but It is not working.

Which code I should follow?

[passing2961]

Hi,

We just create word2count.pkl file by executing nidf.py file in the original repository with a slight change as follows:

print('Counting words in Empathetic Dialogues train set...')
opt['datatype'] = 'train:ordered'
opt['task'] = task
opt['train_experiencer_only'] = True
wc1, ns1 = get_word_counts(opt, count_inputs=True)

print('Counting words in Empathetic Dialogues val set...')
opt['datatype'] = 'valid'
opt['task'] = task
opt['train_experiencer_only'] = True
wc2, ns2 = get_word_counts(opt, count_inputs=True)

As you can see in the above code, we did not convert the empathetic dialogue dataset into the ParlAI format. We just read the dataset and tokenize each utterance based on the whitespace.

Thanks, Young-Jun

[GT-KIM]

Hello,

We solved the problem by changing the data loader for empathetic dialogue data. Unfortunately, the NIDF score is not reproduced even if other scores are reproduced.

Anyway, thank you for your reply!

Best Regards, Gwantae Kim

[passing2961]

Hi,

I'm glad to hear that you've resolved it!

One possible reason for the NIDF score not being reproduced could be due to the stochastic sample of GPT-3 or BlenderBot, which might result in varying NIDF scores. Not only NIDF, but also try other metrics related to the Diversity, such as Entropy-based.

Nevertheless, it's fortunate that other scores have been restored and the issue has been resolved.

I will close this issue.

Best Regards, Young-Jun Lee