Could your publish your code for the processed data？

[ChuanMeng]

I would appreciate it if you could share the code for the the processed data, thanks!

Or could you explain how you do that? For example, what NLP tool did you use? Spacy or nltk?; what is the meaning of "\"", "\< SEP \>" in kowledge and context? and so on. I hope you can give a detailed explanation.

[lizekang]

We use the NLP tools in the OpenNMT-py. You can see the instructions in http://opennmt.net/OpenNMT-py/extended.html. We use tools/tokenizer.perl as tokenizer. """ will be """ after tokenizing and "" will be "< SEP >". And we use to separate multi-turn context.

[ChuanMeng]

Thank you. I understand the function of "< SEP >". But why you convert """ to "\"", beacuse the tokenizer.perl?

[lizekang]

Yes, tokenizer.perl convert """ to """.