NodeLocationsForWays uses huge amount of memory

[breyerml]

We just tried to read a germany osm map using the handler for NodeLocationsForWays

// The type of index used. This must match the include file above
using index_type = osmium::index::map::FlexMem<osmium::unsigned_object_id_type, osmium::Location>;
// The location handler always depends on the index type
using location_handler_type = osmium::handler::NodeLocationsForWays<index_type>;
// The index to hold node locations.
index_type index;
// The location handler will add the node locations to the index and then to the ways
location_handler_type location_handler{index};

as suggested as in one of your examples. But using this handler our memory consumption was increased by 8GB.

After removing this handler and implementing a work around our memory consumption dropped to about 5GB.

Should this be considered normal? I mean 8GB are a lot for such a "small" map.

[Nakaner]

1.0 billion node locations (2 4 byte) fit into an 8 GB large array. If you store the node ID in addtion, 500,000,00 node locations (2 4 byte + 8 byte) fit into 8 GB. Germany currently has 300,997,586 nodes.

An OSM file contains all nodes first, then all ways. Ways don't have node coordinates but references to node IDs only. If you use a NodeLocationsForWays, the node() callback will be called for each node in the input file. Its location will be stored and allows fast access to create geometries when reading ways and relations. They occur later in the file.