[BUG] High memory consumption

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.

https://opensearch.org/docs/latest/opensearch/index/

Apache License 2.0

9.6k stars 1.76k forks source link

[BUG] High memory consumption #15934

Open Mavtti opened 2 weeks ago

Mavtti commented 2 weeks ago

Describe the bug

We are using aws opensearch and since upgrading from 2.11 to 2.15, the memory usage increased and it slows down the whole cluser. The biggest impact is on write operations which started to timeout after the upgrade.

I've seen this issue but I thought that the issue was supposed to be fixed in 2.15.

Upgrade happened on the 10th.

Max memory utilization more than often reaches 100% since then.

There is also an increase in Java Heap memory

Any idea ? Thx !

Related component

Other

To Reproduce

Upgrade to 2.15

Expected behavior

No issue when inserting data and lower memory consumption

Additional Details

No modification on our side on memory settings

reta commented 2 weeks ago

@Mavtti by any chance, could you capture class histogram to understand where the heap consumption is coming from? Thank you.

Mavtti commented 1 week ago

@reta Sorry I never had to do it before, would you mind explaining me how to do it ?

Btw, to mitigate the issue, we changed our instances type from t3 to m7g. It gave us more leeway. But at the same time, we also saw an issue with an ever increasing memory just like this case.

reta commented 1 week ago

@reta Sorry I never had to do it before, would you mind explaining me how to do it ?

Thanks @Mavtti , there are multiple ways to do that (we just need to do this on any node that consumes more heap):

using jcmd (https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr006.html)
```
jcmd <pid> GC.class_histogram
```
using jmap (https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr014.html)
```
jmap  -histo <pid>
```

The Jackson issue should be fixed in 2.15.0 (and above)

Mavtti commented 1 week ago

Hey since I'm on managed aws opensearch, there is no way for me to run these commands.

And I saw that the jackson issue was supposed to be fixed but we actually are on 2.15 (OpenSearch_2_15_R20240904 to be exact) and still see this issue.

reta commented 1 week ago

Hey since I'm on managed aws opensearch, there is no way for me to run these commands.

Got it :(

And I saw that the jackson issue was supposed to be fixed but we actually are on 2.15 (OpenSearch_2_15_R20240904 to be exact) and still see this issue.

Correct, the heap consumption might be caused by another issue

Mavtti commented 1 week ago

So in the end, I can't give you more insight and there's not much you can do without it, correct ?

reta commented 1 week ago

So in the end, I can't give you more insight and there's not much you can do without it, correct ?

I think the best option we have is to engage with AWS support to ask for these details

Mavtti commented 1 week ago

Ok thx @reta for the answers !