Open Dongximing opened 3 years ago
Hi @Dongximing , could you please provide additional details, for example using sample code snippets? How are you using 'build_vocab_from_iterator' together with 'ngrams_iterator'? Also would be great if you can post any error encountered. Thanks!
def yield_tokens(data_iter, ngrams):
for _, text in data_iter:
yield ngrams_iterator(tokenizer(text), ngrams)
I just wondering that 'ngrams_iterator' has become an iterator, but 'yield' also make 'ngrams_iterator(tokenizer(text), ngrams)' become a iterator again. That is why I called "yield of yield "
when I add 'list function'-> list(ngrams_iterator(tokenizer(text), ngrams)) is the same result as without list.
so I mean 'ngrams_iterator' has become an iterator, should we make it to be list and then use yield to make them become a iterator ?
@parmeet
❓ Questions and Help
Description
Hi, I saw the document of torchText, function 'ngrams_iterator' has become a yield, when you call 'build_vocab_from_iterator ' you still have 'yield ngrams_iterator' that become 'yield of yield '? it that ok? thanks