The current implementation of Iterator for Deserializer created by from_reader constructor is eager. Upon calling next() for the first time, it uses Loader to load all the contents via read_to_end() on reader.
Such implementation is unfortunate for two reasons:
it does not allow for online processing of documents on readers without EOF
it wastes space and time, e.g. if someone needs to load only a single or few first documents from a large reader
I suggest loading only a single document from the reader at the time when calling next(). It could replace the current implementation or be optional feature accessible only by, for example, from_reader_lazy() constructor
The current implementation of
IteratorforDeserializercreated byfrom_readerconstructor is eager. Upon callingnext()for the first time, it uses Loader to load all the contents viaread_to_end()on reader.Such implementation is unfortunate for two reasons:
I suggest loading only a single document from the reader at the time when calling
next(). It could replace the current implementation or be optional feature accessible only by, for example,from_reader_lazy()constructor