Closed Joxit closed 3 years ago
Ok, I did some test yesterday with a full WOF only planet import, this is not convincing
master | PR | |
---|---|---|
Size | 3,2G | 48G |
Time | 41m6,438s | 336m20,755s |
I did another test today, save multi-lang names in a field ${placetype}_names
but this causes an error
[400] type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded
Defining too many fields in an index can lead to a mapping explosion, which can cause out of memory errors and difficult situations to recover from.
Consider a situation where every new document inserted introduces new fields, such as with dynamic mapping. Each new field is added to the index mapping, which can become a problem as the mapping grows.
Basically, this is not a good idea, so I will close this PR and find another solution
Ah yeah, adding this many new names probably won't work. There are also issues with field length that start happening with aliases (https://github.com/pelias/pelias/issues/862).
HOWEVER! It might be interesting for you to give it a shot on Elasticsearch 7.10: it has both faster indexing and more space efficient indices. It probably won't make much of a difference but I'm sure happy to be surprised. :)
Yes you're right :disappointed:
I did my tests with Elasticsearch 7.10 :sweat_smile:
Background
With pelias/api#1300 I added some cool feature for i18n autocomplete. But there are some limitations, such as the search of
Parijs, Frankrijk
, due to hierarchy index limitation. Since we have pelias/whosonfirst#446 and the support for multi-lang names property in OSM, we can search for both OSM venues/addresses and WOF by their non-default/English name.This means that if we do an autocomplete of
Parijs
(withlang=nl
and pelias/api#1300) we will get Paris :+1: but if you do an autocomplete ofParijs, Frankrijk
we will get no results :-1:.Why ?
Because the API will search a document with a name property
Parijs
AND in its hierarchy (country, locality...)Frankrijk
. But in our ES index, we only have the default name in document hierarchy. SoFrankrijk
is not in theParijs
hierarchy... OnlyFrance
is present.How to get rid of this ?
We do not have many solutions... I found only 2.
Pont Neuf, Parijs, Frankrijk
:source=wof AND (name.lang='Parijs' or name.lang='Frankrijk')
and take their ids.Pont Neuf
which is in the hierarchy found lately, something likename.lang='Pont Neuf' AND (country.id IN ('whosonfirst:country:85633147', 'whosonfirst:locality:101751119') OR locality IN ('whosonfirst:country:85633147', 'whosonfirst:locality:101751119'))
In this PR I tried the first solution and here are my results for France:
All names in
parent.placetype
First try with all names in
parent.placetype
without duplicates.Parijs, Frankrijk
works with pelias/api#1300 andlang=nl
This first test is not the right one in my opinion. I will try some alternatives :smile:
related: pelias/api#1296 related: pelias/api#1300