search

18F / open-data-maker

make it easy to turn a lot of potentially large csv files into easily accessible open data
Other
199 stars 135 forks source link

Use ElasticSearch bulk import to speed up indexing #263

Closed yozlet closed 8 years ago

yozlet commented 8 years ago

https://www.elastic.co/guide/en/elasticsearch/reference/1.4/docs-bulk.html http://www.rubydoc.info/gems/elasticsearch-api/Elasticsearch/API/Actions#bulk-instance_method

related:

pkarman commented 8 years ago

after some profiling, I do not think ES bulk import would improve the ES speed. 80% of indexing time is spent in marshaling the CSV data.

See #295