search

kohesive / solr-undertow

Solr / SolrCloud running in high performance server - tiny, fast startup, simple to configure, easy deployment without an application server.
Other
96 stars 14 forks source link

Look at tuning options / scripts provided by user #25

Closed apatrida closed 8 years ago

apatrida commented 9 years ago

Look at incorporating as examples, or documentation these scripts provided by user @magicdude4eva

https://gist.github.com/magicdude4eva/3b5fec150fbcaafdc34c

apatrida commented 9 years ago

For the GC tuning, would be good to know what JDK version they are targeted at, assuming JDK 7. How did that compare to G1 without tuning parameters? In the example the heap is 3G, surprised that it has a lot of GC pause or timeout that would need tuned outside of using G1.

apatrida commented 9 years ago

Including a pointer to the GIST in the startup/shutdown section for now, until i can circle back

magicdude4eva commented 9 years ago

The Lucene guys (https://wiki.apache.org/lucene-java/JavaBugs) do not recommend G1:

"Do not, under any circumstances, run Lucene with the G1 garbage collector. Lucene's test suite fails with the G1 garbage collector on a regular basis, including bugs that cause index corruption."

I will take on production load next week - our index is only 1.5GB big and the JVM tuning was based on JDK1.8. Only thing I did was remove incompatible options. I don't feel comfortable using G1 based on the Lucene feedback.

apatrida commented 9 years ago

It looks to be only a 32bit problem for GC1.

magicdude4eva commented 9 years ago

Saw that now too. I think I will try out G1 on JDK8 - for the time being I will use Shawn's baseline settings (unless you have better suggestions) - https://wiki.apache.org/solr/ShawnHeisey

apatrida commented 9 years ago

No, that's a good start

magicdude4eva commented 9 years ago

I updated the GIST with G1 tuning options and have been running it in a non-production environment for 4 days. The JVM settings are based on two indices (2GB and 400MB) running on:

Using GCViewer and JClarity, heap usage is good without any major pauses. Going to update to solr_undertow 1.4 and then switch into production.

magicdude4eva commented 8 years ago

For what it's worth, we could not get the same performance out of Solr running on 5.3.x. We had to revert back to Solr 5.2.1 which outperforms 5.3.x. Over the next week we will move to Solr 5.4.0 and see if this makes a difference.

Software stack is still CentOS7, 4 cores, 10GB RAM. Using JDK 8 64bit 1.8.0_60 with G1GC. Solr heap is set at 3GB with 250ms pause time. This configuration works perfectly fine on 5.2.1.

On 5.3.x Solr runs fine for a number of days and eventually performance degrades. There are no system issues (swapping, IO, spiking CPU) or Java issues (no full GCs). It looks like Solr is just locking up processing requests.

stephlag commented 8 years ago

Have you tried more recent releases (5.5.2 or even 6.1) ?