High memory usage

[isavita]

My application is using ExAws for Lambda call with relatively large body in the response (400KB gzip) responses from aws lambda. I notice that in the request module ExAws.Request there is some copying of the response_body in for telemetry stats here. Is there any way to avoid this (e.g. config)?

Environment

• Elixir & Erlang versions (elixir --version): Elixir 1.14.3 (compiled with Erlang/OTP 23)
• ExAws version: ex_aws 2.4.4
• HTTP client version: finch 0.13.0

Current behavior

There is some copying the response body for :telemetry that increase the memory usage here.

Expected behavior

I would expect to be able to disable this copy of the response body to telemetry or in general to be able to disable telemetry.

[bernardd]

I'm not quite clear how you come to the conclusion that that line causes a copy of the body (not saying it's not, just that I don't understand). Telemetry operations all occur in-process, so unless the body is modified I don't see any reason to imaging it's being copied there.

[isavita]

The flow looks to me as

1. We use this telemetry wrapper that spawn a new :telemetry.span(telemetry_event, telemetry_metadata, fn -> <INNER_SCOPE> end)
2. We do in the http client call to lambda and get result = config[:http_client].request(...). This result has the body.
3. After that we get stop_metadata = %{result: :ok, response_body: Map.get(resp, :body)}. Which is the place that we get the body again.
4. Then we have this part telemetry_metadata = Map.merge(telemetry_metadata, stop_metadata), which I think makes copy of the body.
5. We return {result, telemetry_metadata} in which we have the result with body and telemetry_metadata with the body again.

Am I wrong to think that we copy the body in telemetry_metadata when we merge the metadata? In my application I can see replacing the line stop_metadata = %{result: :ok, response_body: Map.get(resp, :body)} with stop_metadata = %{result: :ok, response_body: ""} decreases the memory usage.

[bernardd]

To the best of my understanding, step 4 should not create a copy of the body - it should just add a reference to the existing binary. How are you measuring memory use? It would be interesting to know exactly what aspect of memory use is increasing (using :erlang.memory() will show you the breakdown) - I think the body would be part of binary so you'd expect to see that jump up at that point if it's really being copied.

[isavita]

@bernardd I've investigated more closely but cannot reproduce the issue. I am closing it.