search

mapbox / osm-wayback

Scalable RocksDB index from OSM planet to lookup historic OSM objects.
BSD 3-Clause "New" or "Revised" License
2 stars 2 forks source link

Lookup should be a Number #5

Closed lukasmartinelli closed 7 years ago

lukasmartinelli commented 7 years ago

I think will be better than storing it as string so it results in faster lookups. In this case we only support a max of 999 versions.

type is either (1=node, 2=way, 3=relation)

osm_id * 10000 + type * 1000 + version

so OSM way with ID 521 and version 120 will result in 5212120. That's better than "way!521!120"

jenningsanderson commented 7 years ago

This seems smart. Is there a noticeable difference in performance? 999 versions max is fine by me; will need to perhaps consider the edge case where we do have > 1000 versions, perhaps just continually over-write the 999th version as the newest?

jenningsanderson commented 7 years ago

@lukasmartinelli Not entirely sure why, but this was failing.

I changed the function to : osm_id + type * 1000 + version and it works great now; haven't done any performance testing, but there are now test files and a few run scripts to assist.

Also, deleted_tags, modified_tags, and new_tags should all be working now.

jenningsanderson commented 7 years ago

Whoops, I just realized why this won't work. Ha. The *10000 padded for the version Number; reopening this issue. We'll need some type of new solution; Would we lose performance if we put nodes ways, and relations into separate DBs?

lukasmartinelli commented 7 years ago

I actually couldn't prove that this speeds it up. But this might be that the "Bloom filter" config doesn't really work. This is supposed to speed up lookups 100x by trading of memory to keep the first 10 bits of a key in memory.