the first arg is specified as an Iterable and the description says "Must yield list or iterator of tokens." It actually wants a list/iterator of lists/iterators of tokens. I guess it's expecting text divided into sentences or something. You can figure out what it's doing easily enough from either the example or the source, but it's unnecessarily confusing.
📚 Documentation
Description
In the docs for build_vocab_from_iterator:
https://pytorch.org/text/stable/vocab.html#build-vocab-from-iterator
the first arg is specified as an
Iterableand the description says "Must yield list or iterator of tokens." It actually wants a list/iterator of lists/iterators of tokens. I guess it's expecting text divided into sentences or something. You can figure out what it's doing easily enough from either the example or the source, but it's unnecessarily confusing.