With these changes (elaborated in #3), I can process planet.osm.pbf (40+GB) and begin receiving results right away while using much less memory than before. I tried to keep this minimal, but I did need to add Send to the traits so that par_bridge() would work for the par_map_reduce() method.
I think a bigger change which I mentioned in #3 already would be to have .iter() and .par_iter() methods with the relevant decoding logic attached so that you could use any iteration methods which the std or rayon defines, but this patch is designed to be minimal and only fixes the existing methods so that they will work on large files without buffering into memory.
With these changes (elaborated in #3), I can process planet.osm.pbf (40+GB) and begin receiving results right away while using much less memory than before. I tried to keep this minimal, but I did need to add
Sendto the traits so thatpar_bridge()would work for thepar_map_reduce()method.I think a bigger change which I mentioned in #3 already would be to have
.iter()and.par_iter()methods with the relevant decoding logic attached so that you could use any iteration methods which the std or rayon defines, but this patch is designed to be minimal and only fixes the existing methods so that they will work on large files without buffering into memory.