search

allenai / bilm-tf

Tensorflow implementation of contextualized word representations from bi-directional language models
Apache License 2.0
1.62k stars 452 forks source link

Is it necessary (!,?\,,\(\)) in Vocab ? #155

Closed edunatalec closed 5 years ago

edunatalec commented 5 years ago

I'm creating a code to create vocab. But I don't know if the signals is really necessary in vocab.

PhilipMay commented 5 years ago

I am not sure if I understand your question. Is your question is if characters like !, ? and , should be part of the vocabulary?

If this is your question the answer is: definitely yes.