This might be rather curious or anecdotal, but maybe someone else stumbles over this too:
I created a custom OSM file from proprietary data (no mingling with actual OSM data). Most objects have their own IDs already which I just use as OSM ID (weird range from 1 - 48ish bits). Some objects don't and I have to assign an ID. Since OSM IDs are technically valid up to 64bit and this custom dataset doesn't seem to have IDs > 50ish bits, I went up to 62bit for the "synthetic" IDs.
When importing them into ES it seemed to be ok according to the logs:
log
info: [openstreetmap] Creating read stream for: /data/openstreetmap/berlin-charging.eclever.pbf
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/ASM.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/GUM.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/MNP.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/PRI.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/USA.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/USA.override.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pe
[berlin-charging.eclever.pbf.zip](https://github.com/pelias/docker/files/9315509/berlin-charging.eclever.pbf.zip)
lias-wof-admin-lookup/src/data/VIR.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/NLD.tsv
info: [wof-pip-service:master] starting with layers neighbourhood,borough,locality,localadmin,county,macrocounty,macroregion,region,dependency,country,empire,continent,marinearea,ocean
info: [wof-pip-service:master] borough worker loaded 12 features in 0.031 seconds
info: [wof-pip-service:master] locality worker loaded 1 features in 0.02 seconds
info: [wof-pip-service:master] county worker loaded 0 features in 0.014 seconds
info: [wof-pip-service:master] localadmin worker loaded 1 features in 0.023 seconds
info: [wof-pip-service:master] macrocounty worker loaded 0 features in 0.011 seconds
info: [wof-pip-service:master] macroregion worker loaded 0 features in 0.01 seconds
info: [wof-pip-service:master] neighbourhood worker loaded 103 features in 0.106 seconds
info: [wof-pip-service:master] region worker loaded 1 features in 0.024 seconds
info: [wof-pip-service:master] dependency worker loaded 0 features in 0.011 seconds
info: [wof-pip-service:master] empire worker loaded 0 features in 0.009 seconds
info: [wof-pip-service:master] continent worker loaded 0 features in 0.009 seconds
info: [wof-pip-service:master] marinearea worker loaded 0 features in 0.01 seconds
info: [wof-pip-service:master] ocean worker loaded 0 features in 0.01 seconds
info: [wof-pip-service:master] country worker loaded 1 features in 0.072 seconds
info: [wof-pip-service:master] PIP Service Loading Completed!!!
info: [wof-admin-lookup] Shutting down admin lookup service
info: [admin-lookup:worker] neighbourhood worker process exiting, stats: {"calls":3,"hits":3,"misses":0}
info: [admin-lookup:worker] ocean worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] macrocounty worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] country worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] borough worker process exiting, stats: {"calls":3,"hits":3,"misses":0}
info: [admin-lookup:worker] localadmin worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] dependency worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] region worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] empire worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] continent worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] marinearea worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] locality worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] county worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] macroregion worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [dbclient-openstreetmap] paused=false, transient=1, current_length=0
info: [dbclient-openstreetmap] paused=false, transient=1, current_length=0
info: [dbclient-openstreetmap] paused=false, transient=0, current_length=0, indexed=3, batch_ok=1, batch_retries=0, failed_records=0, venue=3, persec=0.3
info: [dbclient-openstreetmap] paused=false, transient=0, current_length=0, indexed=3, batch_ok=1, batch_retries=0, failed_records=0, venue=3, persec=0.3
But then I couldn't find them with /search, e.g. /v1/search?categories=transport&focus.point.lat=52.469763&focus.point.lon=13.440254&text=bla1 and the pelias elastic stats says:
Either the importer complaining about "invalid" entries or a successful import. Like I said, curious but pretty niche. And it did work with up to 50bit range (1e15).
Environment (please complete the following information):
Describe the bug
This might be rather curious or anecdotal, but maybe someone else stumbles over this too:
I created a custom OSM file from proprietary data (no mingling with actual OSM data). Most objects have their own IDs already which I just use as OSM ID (weird range from 1 - 48ish bits). Some objects don't and I have to assign an ID. Since OSM IDs are technically valid up to 64bit and this custom dataset doesn't seem to have IDs > 50ish bits, I went up to 62bit for the "synthetic" IDs.
When importing them into ES it seemed to be ok according to the logs:
log
info: [openstreetmap] Creating read stream for: /data/openstreetmap/berlin-charging.eclever.pbf debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/ASM.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/GUM.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/MNP.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/PRI.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/USA.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/USA.override.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pe [berlin-charging.eclever.pbf.zip](https://github.com/pelias/docker/files/9315509/berlin-charging.eclever.pbf.zip) lias-wof-admin-lookup/src/data/VIR.tsv debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/NLD.tsv info: [wof-pip-service:master] starting with layers neighbourhood,borough,locality,localadmin,county,macrocounty,macroregion,region,dependency,country,empire,continent,marinearea,ocean info: [wof-pip-service:master] borough worker loaded 12 features in 0.031 seconds info: [wof-pip-service:master] locality worker loaded 1 features in 0.02 seconds info: [wof-pip-service:master] county worker loaded 0 features in 0.014 seconds info: [wof-pip-service:master] localadmin worker loaded 1 features in 0.023 seconds info: [wof-pip-service:master] macrocounty worker loaded 0 features in 0.011 seconds info: [wof-pip-service:master] macroregion worker loaded 0 features in 0.01 seconds info: [wof-pip-service:master] neighbourhood worker loaded 103 features in 0.106 seconds info: [wof-pip-service:master] region worker loaded 1 features in 0.024 seconds info: [wof-pip-service:master] dependency worker loaded 0 features in 0.011 seconds info: [wof-pip-service:master] empire worker loaded 0 features in 0.009 seconds info: [wof-pip-service:master] continent worker loaded 0 features in 0.009 seconds info: [wof-pip-service:master] marinearea worker loaded 0 features in 0.01 seconds info: [wof-pip-service:master] ocean worker loaded 0 features in 0.01 seconds info: [wof-pip-service:master] country worker loaded 1 features in 0.072 seconds info: [wof-pip-service:master] PIP Service Loading Completed!!! info: [wof-admin-lookup] Shutting down admin lookup service info: [admin-lookup:worker] neighbourhood worker process exiting, stats: {"calls":3,"hits":3,"misses":0} info: [admin-lookup:worker] ocean worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] macrocounty worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] country worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] borough worker process exiting, stats: {"calls":3,"hits":3,"misses":0} info: [admin-lookup:worker] localadmin worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] dependency worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] region worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] empire worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] continent worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] marinearea worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] locality worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] county worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [admin-lookup:worker] macroregion worker process exiting, stats: {"calls":0,"hits":0,"misses":0} info: [dbclient-openstreetmap] paused=false, transient=1, current_length=0 info: [dbclient-openstreetmap] paused=false, transient=1, current_length=0 info: [dbclient-openstreetmap] paused=false, transient=0, current_length=0, indexed=3, batch_ok=1, batch_retries=0, failed_records=0, venue=3, persec=0.3 info: [dbclient-openstreetmap] paused=false, transient=0, current_length=0, indexed=3, batch_ok=1, batch_retries=0, failed_records=0, venue=3, persec=0.3But then I couldn't find them with
/search
, e.g./v1/search?categories=transport&focus.point.lat=52.469763&focus.point.lon=13.440254&text=bla1
and thepelias elastic stats
says:I guess it's some ES field definition (no idea about ES :D)?
Steps to Reproduce Load the attached minimal PBF file:
Expected behavior
Either the importer complaining about "invalid" entries or a successful import. Like I said, curious but pretty niche. And it did work with up to 50bit range (1e15).
Environment (please complete the following information):