search

elastic / stream2es

Stream data into ES (Wikipedia, Twitter, stdin, or other ESes)
355 stars 60 forks source link

avoid holding full stream in memory #15

Closed MichaelBlume closed 10 years ago

dakrone commented 10 years ago

This looks good to me, @drewr?

grmblfrz commented 10 years ago

Without this patch, reindexing 6 million documents could not finish - not enough memory. Please integrate this patch.

drewr commented 10 years ago

Yikes! Happened to see this due to @grmblfrz notification. Sorry for the delay and thanks @MichaelBlume! That doall wasn't supposed to stay in there.

arowla commented 10 years ago

Has this been released to download.elasticsearch.org yet? I am getting out of memory errors when trying to use stream2es es.

EDIT: Nevermind, I built from source and still got the same error. I just submitted issue #24 about this.