search

matthaywardwebdesign / aus-search

A collection of Node JS scripts to create an Elasticsearch index of Australian addresses.
MIT License
81 stars 20 forks source link

Geospatial indexes #1

Closed nikolayg closed 7 years ago

nikolayg commented 7 years ago

First of all, this is a great tool - thanks for building it.

I was looking at config/indexes.json and I couldn't see any geospatial indexes. Any guidance on how to quickly add geo indexes would be appreciated!

matthaywardwebdesign commented 7 years ago

@nikolayg I'm guessing you were planning on querying the data using MongoDB then? :)

nikolayg commented 7 years ago

Ah yes, in Mongo. I tweaked your code a little bit and managed to get 2dsphere indexes.

On another note - there's an issue when you try to populate a remote mongo server over the network. The scripts read the files and populate the buffers much faster than the DB insertion. This results in out-of-memory exceptions. Even 4G of --max-old-space-size didn't help. I ended up running the scripts with a local Mongo, and then imported the dumps in the server.

matthaywardwebdesign commented 7 years ago

Ah nice work!

Thanks for discovering that issue, probably should add an option to wait for the records to be inserted before reading more from the files, that should eliminate the issue.