large jsonl/ndjson-files create OOM-error

[pekasen]

If lines is set to True for the Twacapic-Reader, the reader tries to read the file all at once, thus, creating an OOM-error when the file is sufficiently large.

[pekasen]

following snippet does the trick:

with Path("large_file_to_open.dat").open("r", encoding="utf8") as file:
  for line in file:  
    yield ujson.loads(line)

This reduces the memory consumption from as much as the file needs to just the memory footprint of each line.