Closed Joxit closed 4 years ago
I remove the ORDER BY
query, this does not fit on world database. Instead I use the combined stream and filter on placetypes.
Oh yes, and with the new change, it does not decrease import time on France imports and works better on world imports ! :tada:
Oops, we forgot about this PR which looks very nice! We will be doing a round of work with SQLite WOF import soon, and will take a look at this! Thanks for making this PR :)
Looks great, and this does indeed fix a bunch of issues when importing WOF data.
I made one tiny change to move the logger output from info
level to debug
, but otherwise this is great! thanks!
:wave: I did some awesome work for the Pelias project and would love for everyone to have a look at it and provide feedback.
Here's the reason for this change :rocket:
Some WOF data hierarchies were broken due to the import order and how
resolveHierarchy
work. If you have a recordB
which has in its hierarchyA
andC
. ThenC
will be missing because not yet imported.This issue is linked to autocomplete only, because search use the hierarchy from placeholder and not ES.
For example, Tavel in France, when I import all France database, the hierarchy is :
Here's what actually got changed :clap:
I added an order by clause in the SQL statement (optional). It's use only when we do an import in the
readStream
, that meas wof admin lookup will not be impacted.Now we have the correct result.
Here's how others can test the changes :eyes:
There are some tests, If you remove the option in
readStream
you will see the issue. This will also slow down import time... :/ The import time for France has increased from 1m 25s to 2m 28s with this PR... :/