search

crate / elasticsearch-inout-plugin

An Elasticsearch plugin which provides the ability to export data by query on server side.
Apache License 2.0
112 stars 15 forks source link

Compatibility with 0.90.3 #35

Open gkarlikatwork opened 11 years ago

gkarlikatwork commented 11 years ago

Hi,

due to changes on elasticsearch side in 0.90.3 plugin does not build anymore.

mfussenegger commented 11 years ago

Hi thanks for the report.

We're aware that the recent update broke some of our plugin. Our plan was to finish up some other work before we start updating the plugins.

For this plugin I've already created a pull request here https://github.com/crate/elasticsearch-inout-plugin/pull/36. It builds again and tests pass. But I will have to follow up with a real world test (hopefully tomorrow) before I can merge it.

So we should have a new release out soon enough.

gkarlikatwork commented 11 years ago

Hi,

thanks for your checkin. I tried it and start restoring my test full dump. I got immediately FullGCs in Elasticseach. I had no problems with same data in 0.90.1. I use default elasticsearch GC Settings with 1GB Xmx. I tried with 1.5 Gb Xmx but it does not solve this problem. My test dump has 12 millions docs with 600 MB of json files ...

mfussenegger commented 11 years ago

I've added another commit to the 0.90.3 branch that should help reduce the GCs see https://github.com/crate/elasticsearch-inout-plugin/tree/0.90.3

Can you try that and see if it helps? How exactly are the GCs affecting you? Is a performance difference noticeable between 0.90.1 and 0.90.3?

gkarlikatwork commented 11 years ago

Hi,

the performance is really bad with 0.90.3. But i think, it's because of FullGCs ... I will try you checkin and post a result.

gkarlikatwork commented 11 years ago

I've tried your checkin, but unfortunately, the same problem still exists.

My GC Log looks like this: 2013-08-29T11:30:19.740+0200: 44,469: [Full GC 44,469: [CMS2013-08-29T11:30:19.769+0200: 44,498: [CMS-concurrent-mark: 0,040/0,042 secs] [Times: user=0,15 sys=0,01, real=0,04 secs](concurrent mode failure): 699068K->699071K(699072K), 0,2804530 secs] 1013627K->802848K(1013632K), [CMS Perm : 35901K->35901K(59884K)], 0,2805270 secs] [Times: user=0,32 sys=0,00, real=0,28 secs] 2013-08-29T11:30:20.029+0200: 44,758: [GC [1 CMS-initial-mark: 699071K(699072K)] 825226K(1013632K), 0,0080890 secs] [Times: user=0,01 sys=0,00, real=0,01 secs] 2013-08-29T11:30:20.038+0200: 44,767: [CMS-concurrent-mark-start] 2013-08-29T11:30:20.109+0200: 44,838: [Full GC 44,838: [CMS2013-08-29T11:30:20.113+0200: 44,842: [CMS-concurrent-mark: 0,069/0,076 secs] [Times: user=0,53 sys=0,00, real=0,08 secs](concurrent mode failure): 699071K->699044K(699072K), 0,2318230 secs] 1013609K->867754K(1013632K), [CMS Perm : 35904K->35904K(59884K)], 0,2318940 secs] [Times: user=0,24 sys=0,00, real=0,24 secs] 2013-08-29T11:30:20.394+0200: 45,123: [Full GC 45,123: [CMS: 699044K->699043K(699072K), 0,2267800 secs] 1013587K->907125K(1013632K), [CMS Perm : 35907K->35907K(59884K)], 0,2268510 secs] [Times: user=0,25 sys=0,00, real=0,23 secs] 2013-08-29T11:30:20.622+0200: 45,351: [GC [1 CMS-initial-mark: 699043K(699072K)] 909525K(1013632K), 0,0070420 secs] [Times: user=0,01 sys=0,00, real=0,00 secs] 2013-08-29T11:30:20.629+0200: 45,358: [CMS-concurrent-mark-start] 2013-08-29T11:30:20.684+0200: 45,413: [Full GC 45,413: [CMS2013-08-29T11:30:20.689+0200: 45,418: [CMS-concurrent-mark: 0,057/0,060 secs] [Times: user=0,41 sys=0,00, real=0,06 secs](concurrent mode failure): 699050K->699050K(699072K), 0,1611840 secs] 1013610K->936892K(1013632K), [CMS Perm : 35908K->35908K(59884K)], 0,1612560 secs] [Times: user=0,17 sys=0,00, real=0,16 secs] 2013-08-29T11:30:20.858+0200: 45,587: [GC [1 CMS-initial-mark: 699050K(699072K)] 984412K(1013632K), 0,0157220 secs] [Times: user=0,02 sys=0,00, real=0,02 secs] 2013-08-29T11:30:20.874+0200: 45,603: [CMS-concurrent-mark-start] 2013-08-29T11:30:20.905+0200: 45,634: [Full GC 45,634: [CMS2013-08-29T11:30:20.929+0200: 45,658: [CMS-concurrent-mark: 0,049/0,055 secs] [Times: user=0,26 sys=0,01, real=0,05 secs](concurrent mode failure): 699050K->699041K(699072K), 0,2526990 secs] 1013581K->964362K(1013632K), [CMS Perm : 35913K->35913K(59884K)], 0,2527690 secs] [Times: user=0,29 sys=0,00, real=0,25 secs] 2013-08-29T11:30:21.190+0200: 45,919: [Full GC 45,919: [CMS: 699041K->699041K(699072K), 0,1645230 secs] 1013592K->983925K(1013632K), [CMS Perm : 35916K->35916K(59884K)], 0,1645950 secs] [Times: user=0,17 sys=0,00, real=0,17 secs] 2013-08-29T11:30:21.355+0200: 46,084: [GC [1 CMS-initial-mark: 699041K(699072K)] 983985K(1013632K), 0,0078270 secs] [Times: user=0,01 sys=0,00, real=0,01 secs] 2013-08-29T11:30:21.363+0200: 46,092: [CMS-concurrent-mark-start]

gkarlikatwork commented 11 years ago

This issue could be a problem: https://github.com/elasticsearch/elasticsearch/issues/3638

gkarlikatwork commented 11 years ago

Plugin works fine with 0.90.4

gkarlikatwork commented 11 years ago

Plugin does not build with 0.90.4, 0.90.5. I tried it as a binary from 0.90.3 before and import works well.

apatrida commented 10 years ago

See other tickets, I ported to 0.90.5, 0.90.6 and 1.0.0.beta1 as pull requests.

If you apply the pull requests you can help test.