pszufe / OpenStreetMapX.jl

OpenStreetMap (*.osm) support for Julia 1.0 and up
MIT License
123 stars 24 forks source link

Use concrete types instead of dictionaries #52

Closed blegat closed 3 years ago

blegat commented 3 years ago

As discussed in https://github.com/pszufe/OpenStreetMapX.jl/pull/44#discussion_r667459615 and https://github.com/JuliaIO/ProtoBuf.jl/issues/179, creating the small dictionaries can be a bottleneck when reading many small structs. I assume that for DenseNodes, it it's ok since it's one bit struct but for reading all the small Ways and Relations, it can be quite noticeable. As the timings depends on the % of the GC time, I give a few different runs (I know, I should use BenchmarkTools ^^)

Currently:

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.819850 seconds (1.58 M allocations: 183.134 MiB, 17.31% gc time)

After https://github.com/JuliaIO/ProtoBuf.jl/pull/184:

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.358973 seconds (1.14 M allocations: 176.507 MiB, 16.71% gc time)

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.312807 seconds (1.14 M allocations: 176.506 MiB, 5.27% gc time)

After this PR:

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.743238 seconds (1.01 M allocations: 141.000 MiB, 20.39% gc time)

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.669973 seconds (1.01 M allocations: 141.000 MiB, 12.62% gc time)

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.626401 seconds (1.01 M allocations: 141.000 MiB, 6.59% gc time)

After both PR:

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.266550 seconds (578.92 k allocations: 134.372 MiB, 16.38% gc time)
pszufe commented 3 years ago

Thanks!