pelias / docker

Run the Pelias geocoder in docker containers, including example projects.
MIT License
315 stars 218 forks source link

inherent limit on OSM ID bit range? #297

Open nilsnolde opened 1 year ago

nilsnolde commented 1 year ago

Describe the bug

This might be rather curious or anecdotal, but maybe someone else stumbles over this too:

I created a custom OSM file from proprietary data (no mingling with actual OSM data). Most objects have their own IDs already which I just use as OSM ID (weird range from 1 - 48ish bits). Some objects don't and I have to assign an ID. Since OSM IDs are technically valid up to 64bit and this custom dataset doesn't seem to have IDs > 50ish bits, I went up to 62bit for the "synthetic" IDs.

When importing them into ES it seemed to be ok according to the logs:

log info: [openstreetmap] Creating read stream for: /data/openstreetmap/berlin-charging.eclever.pbf debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/ASM.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/GUM.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/MNP.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/PRI.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/USA.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/USA.override.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pe [berlin-charging.eclever.pbf.zip](https://github.com/pelias/docker/files/9315509/berlin-charging.eclever.pbf.zip) lias-wof-admin-lookup/src/data/VIR.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/NLD.tsv info: [wof-pip-service:master] starting with layers neighbourhood,borough,locality,localadmin,county,macrocounty,macroregion,region,dependency,country,empire,continent,marinearea,ocean info: [wof-pip-service:master] borough worker loaded 12 features in 0.031 seconds info: [wof-pip-service:master] locality worker loaded 1 features in 0.02 seconds info: [wof-pip-service:master] county worker loaded 0 features in 0.014 seconds info: [wof-pip-service:master] localadmin worker loaded 1 features in 0.023 seconds info: [wof-pip-service:master] macrocounty worker loaded 0 features in 0.011 seconds info: [wof-pip-service:master] macroregion worker loaded 0 features in 0.01 seconds info: [wof-pip-service:master] neighbourhood worker loaded 103 features in 0.106 seconds info: [wof-pip-service:master] region worker loaded 1 features in 0.024 seconds info: [wof-pip-service:master] dependency worker loaded 0 features in 0.011 seconds info: [wof-pip-service:master] empire worker loaded 0 features in 0.009 seconds info: [wof-pip-service:master] continent worker loaded 0 features in 0.009 seconds info: [wof-pip-service:master] marinearea worker loaded 0 features in 0.01 seconds info: [wof-pip-service:master] ocean worker loaded 0 features in 0.01 seconds info: [wof-pip-service:master] country worker loaded 1 features in 0.072 seconds info: [wof-pip-service:master] PIP Service Loading Completed!!! info: [wof-admin-lookup] Shutting down admin lookup service info: [admin-lookup:worker] neighbourhood worker process exiting, stats: {"calls":3,"hits":3,"misses":0} info: [admin-lookup:worker] ocean worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] macrocounty worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] country worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] borough worker process exiting, stats: {"calls":3,"hits":3,"misses":0} info: [admin-lookup:worker] localadmin worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] dependency worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] region worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] empire worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] continent worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] marinearea worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] locality worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] county worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] macroregion worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [dbclient-openstreetmap] paused=false, transient=1, current_length=0 info: [dbclient-openstreetmap] paused=false, transient=1, current_length=0 info: [dbclient-openstreetmap] paused=false, transient=0, current_length=0, indexed=3, batch_ok=1, batch_retries=0, failed_records=0, venue=3, persec=0.3 info: [dbclient-openstreetmap] paused=false, transient=0, current_length=0, indexed=3, batch_ok=1, batch_retries=0, failed_records=0, venue=3, persec=0.3

But then I couldn't find them with /search, e.g. /v1/search?categories=transport&focus.point.lat=52.469763&focus.point.lon=13.440254&text=bla1 and the pelias elastic stats says:

        {
          "key" : "openstreetmap",
          "doc_count" : 1,
          "layers" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "venue",
                "doc_count" : 1
              }
            ]
          }
        }

I guess it's some ES field definition (no idea about ES :D)?

Steps to Reproduce Load the attached minimal PBF file:

pelias elastic start && pelias elastic wait && pelias elastic create
pelias import osm
pelias elastic stats

Expected behavior

Either the importer complaining about "invalid" entries or a successful import. Like I said, curious but pretty niche. And it did work with up to 50bit range (1e15).

Environment (please complete the following information):

michaelkirk commented 11 months ago

Just a wild guess, but maybe related to javascript's MAX_SAFE_INTEGER=2^53-1