I'm doing a performance test of an endpoint which is trying to get 10000 hits from ES. The profiling give me a high memory usage. But I am expecting a reasonable memory consumption
Please describe the actual behavior
pprof tool shows me that io.ReadAll is using considerable amount of memory when searching 10K hits from ES. This bound our application to a certain amount of concurrent requests due to the memory overhead. See the following part from the memory profile
I used the io.Copy instead of io.ReadAll and tried the same amount of concurrent requests. This improves the situation significantly. see the profile below
See the chages I made to response.go file below
According to above example, io.ReadAll has an allocated memory space of around 145MB while other has only around 57MB. I think this will significantly improve the memory usage of any application, and this is an issue we are facing as well. Shall I create a PR with this change?
Any steps to reproduce the behavior?
Difference of memory usage between io.Copy vs io.ReadAll can be compared when searching for 10K hits
Please use the following questions as a guideline to help me answer your issue/question without further inquiry. Thank you.
Which version of Elastic are you using?
[X ] elastic.v7 (for Elasticsearch 7.x) [ ] elastic.v6 (for Elasticsearch 6.x) [ ] elastic.v5 (for Elasticsearch 5.x) [ ] elastic.v3 (for Elasticsearch 2.x) [ ] elastic.v2 (for Elasticsearch 1.x) olivere/elastic/v7 --> v7.0.21
Please describe the expected behavior
I'm doing a performance test of an endpoint which is trying to get 10000 hits from ES. The profiling give me a high memory usage. But I am expecting a reasonable memory consumption
Please describe the actual behavior
pprof tool shows me that io.ReadAll is using considerable amount of memory when searching 10K hits from ES. This bound our application to a certain amount of concurrent requests due to the memory overhead. See the following part from the memory profile
I used the io.Copy instead of io.ReadAll and tried the same amount of concurrent requests. This improves the situation significantly. see the profile below
See the chages I made to response.go file below
According to above example, io.ReadAll has an allocated memory space of around 145MB while other has only around 57MB. I think this will significantly improve the memory usage of any application, and this is an issue we are facing as well. Shall I create a PR with this change?
Any steps to reproduce the behavior?
Difference of memory usage between io.Copy vs io.ReadAll can be compared when searching for 10K hits