search

elastic / elasticsearch-cloud-aws

AWS Cloud Plugin for Elasticsearch
https://github.com/elastic/elasticsearch/tree/master/plugins/discovery-ec2
577 stars 180 forks source link

Proliferation of S3ResponseMetadata #193

Closed drewr closed 9 years ago

drewr commented 9 years ago

GitHub has been experiencing an accumulation of S3ResponseMetadata objects. It appears that they linger long enough to get to old gen but then become unattached. For example, in one heap dump, there was 2.1gb of them (compared to 23mb reachable objs).

Is there any way we can avoid the cache that is involved in this call? That seems to be the stem. This is on a 1.2.1 cluster running cloud-aws 2.1.1, however, maybe this has been addressed in newer versions of cloud-aws and aws-sdk?

https://github.com/elastic/elasticsearch-cloud-aws/blob/master/src/test/java/org/elasticsearch/cloud/aws/AmazonS3Wrapper.java#L527

cc @tlrx @dadoonet @rmuir @TwP @grantr

grantr commented 9 years ago

We still see this on our 1.4.2 cluster running cloud-aws 2.4.1.

tlrx commented 9 years ago

I had a look at this issue today but I can't reproduce it. There's no way, even in the more recent version of the AWS SDK, to disable the ResponseMetadataCache use internally by the HTTP client.

After reviewing the code I don't think we keep some object refs around... I first suspected the removeEldestEntry method to leak but cannot confirm this.

cloud-aws 2.4.1 uses AWS SDK v1.7.3 which is a bit old now. Cloud-aws 2.5.0 uses a more recent version of the SDK, I'm curious to know if it has the same issue.

rmuir commented 9 years ago

I submitted a PR for this here: https://github.com/aws/aws-sdk-java/pull/415

drewr commented 9 years ago

Thanks @rmuir!

tlrx commented 9 years ago

@rmuir awesome :+1: