Closed Witiko closed 2 years ago
@dorianbrown Please, let me know if there is anything I can do to get this merged.
Hi Wikito,
Thanks a lot for this contribution, the case of using a generator as a corpus never occurred to me, but seems like a very useful bit of functionality to have. And thanks for you patience, it's been a bit of a busy few weeks :smile:
After looking through the changes this doesn't seem to cause any issues, so I'll merge it in and check if my CD is still working correctly.
The branch has been merged, and the changes have been published to pypi under the version 0.2.2
of the package.
Thanks again for helping make this package better!
@dorianbrown Thank you, much appreciated.
Currently, rank-bm25 requires that
corpus
is repeatedly iterable and sized (i.e. defines__len__()
).When the corpus is not pre-tokenized (i.e.
tokenizer
is not None), then this makes sense:__init__()
will iterate across the corpus several times, so we may as well require that the corpus is a list or some other data type that is repeatedly iterable and sized. However, when the corpus is pre-tokenized (i.e.tokenizer
is None), then we only iterate over the corpus once in_initialize()
. Furthermore, we don't need to know its size beforehand, because we can just count the number of iterations.This pull request makes it possible to use a non-repeatedly iterable non-sized corpus such as a generator when
tokenizer
is None. This is useful if you need to generate your corpus on the fly and don't know the number of your documents beforehand.