search

awslabs / aws-crt-java

Java bindings for the AWS Common Runtime
Apache License 2.0
55 stars 40 forks source link

Seg fault in aws_java_http_stream_on_stream_destroy_fn #801

Open andreasparteli opened 3 months ago

andreasparteli commented 3 months ago

Describe the bug

At a seemingly random time our Java Process dies due to a SIGSEGV in AWS CRT:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f4444dc6b55, pid=1, tid=47
#
# JRE version: OpenJDK Runtime Environment Corretto-11.0.23.9.1 (11.0.23+9) (build 11.0.23+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM Corretto-11.0.23.9.1 (11.0.23+9-LTS, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [AWSCRT_2322636264284720268libaws-crt-jni.so+0x8bb55]  aws_java_http_stream_on_stream_destroy_fn+0x25
#
# Core dump will be written. Default location: /app/core.1
#
# An error report file with more information is saved as:
# /app/hs_err_pid1.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/corretto/corretto-11/issues/
#

It's hard to reproduce as the affected service runs for hours at times executing the same code that later causes the crash.

This was already described in https://github.com/awslabs/aws-crt-java/issues/763. I managed to get a CRT debug log and am still investigating further.

Expected Behavior

No crash

Current Behavior

2024-06-24 12:17:55.469,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-stream] - id=0x7f443de623a0: Client request complete, response status: 200 (OK).
2024-06-24 12:17:55.469,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-connection] - ConnManager Releasing Conn: manager: 0x7f4446adc0a0, conn: 0x7f443de63f50
2024-06-24 12:17:55.469,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: User releasing connection (id=0x7f443de63f50)
2024-06-24 12:17:55.469,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=2, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=2, ref_count=1
2024-06-24 12:17:55.517,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [http-connection] - Requesting a new connection from conn_manager: 0x7f4446adc0a0
2024-06-24 12:17:55.517,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [connection-manager] - id=0x7f4446adc0a0: Acquire connection
2024-06-24 12:17:55.517,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [connection-manager] - id=0x7f4446adc0a0: Grabbing pooled connection (0x7f443de63f50)
2024-06-24 12:17:55.517,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=1, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=2, ref_count=1
2024-06-24 12:17:55.517,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: Successfully completed connection acquisition with connection id=0x7f443de63f50
2024-06-24 12:17:55.517,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-connection] - ConnManager Acquired Conn: conn: 0x7f443de63f50, manager: 0x7f4446adc0a0, err_code: 0,  err_str: Success.
2024-06-24 12:17:55.517,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-stream] - id=0x7f443de62970: Created client request on connection=0x7f443de63f50: POST / HTTP/1.1
2024-06-24 12:17:55.571,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-stream] - id=0x7f443de62970: Client request complete, response status: 200 (OK).
2024-06-24 12:17:55.571,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-connection] - ConnManager Releasing Conn: manager: 0x7f4446adc0a0, conn: 0x7f443de63f50
2024-06-24 12:17:55.571,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: User releasing connection (id=0x7f443de63f50)
2024-06-24 12:17:55.571,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=2, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=2, ref_count=1
2024-06-24 12:17:55.571,[DEBUG] [2024-06-24T12:17:55Z] [00007f444324eb38] [http-connection] - Requesting a new connection from conn_manager: 0x7f4446adc0a0
2024-06-24 12:17:55.571,[DEBUG] [2024-06-24T12:17:55Z] [00007f444324eb38] [connection-manager] - id=0x7f4446adc0a0: Acquire connection
2024-06-24 12:17:55.571,[DEBUG] [2024-06-24T12:17:55Z] [00007f444324eb38] [connection-manager] - id=0x7f4446adc0a0: Grabbing pooled connection (0x7f443de63f50)
2024-06-24 12:17:55.579,[DEBUG] [2024-06-24T12:17:55Z] [00007f444324eb38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=1, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=2, ref_count=1
2024-06-24 12:17:55.579,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: Successfully completed connection acquisition with connection id=0x7f443de63f50
2024-06-24 12:17:55.579,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-connection] - ConnManager Acquired Conn: conn: 0x7f443de63f50, manager: 0x7f4446adc0a0, err_code: 0,  err_str: Success.
2024-06-24 12:17:55.579,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-stream] - id=0x7f443de63320: Created client request on connection=0x7f443de63f50: POST / HTTP/1.1
2024-06-24 12:17:55.579,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-stream] - id=0x7f443de63320: Client request complete, response status: 200 (OK).
2024-06-24 12:17:55.579,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-connection] - ConnManager Releasing Conn: manager: 0x7f4446adc0a0, conn: 0x7f443de63f50
2024-06-24 12:17:55.589,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: User releasing connection (id=0x7f443de63f50)
2024-06-24 12:17:55.589,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=2, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=2, ref_count=1
2024-06-24 12:17:55.589,[DEBUG] [2024-06-24T12:17:55Z] [00007f444324eb38] [http-connection] - Requesting a new connection from conn_manager: 0x7f4446adc0a0
2024-06-24 12:17:55.589,[DEBUG] [2024-06-24T12:17:55Z] [00007f444324eb38] [connection-manager] - id=0x7f4446adc0a0: Acquire connection
2024-06-24 12:17:55.589,[DEBUG] [2024-06-24T12:17:55Z] [00007f444324eb38] [connection-manager] - id=0x7f4446adc0a0: Grabbing pooled connection (0x7f443de63f50)
2024-06-24 12:17:55.589,[DEBUG] [2024-06-24T12:17:55Z] [00007f444324eb38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=1, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=2, ref_count=1
2024-06-24 12:17:55.594,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: Successfully completed connection acquisition with connection id=0x7f443de63f50
2024-06-24 12:17:55.594,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-connection] - ConnManager Acquired Conn: conn: 0x7f443de63f50, manager: 0x7f4446adc0a0, err_code: 0,  err_str: Success.
2024-06-24 12:17:55.594,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-stream] - id=0x7f443de63610: Created client request on connection=0x7f443de63f50: POST / HTTP/1.1
2024-06-24 12:17:55.594,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-stream] - id=0x7f443de63610: Client request complete, response status: 200 (OK).
2024-06-24 12:17:55.594,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-connection] - ConnManager Releasing Conn: manager: 0x7f4446adc0a0, conn: 0x7f443de63f50
2024-06-24 12:17:55.594,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: User releasing connection (id=0x7f443de63f50)
2024-06-24 12:17:55.594,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=2, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=2, ref_count=1
2024-06-24 12:17:55.605,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [http-connection] - Requesting a new connection from conn_manager: 0x7f4446adc0a0
2024-06-24 12:17:55.605,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [connection-manager] - id=0x7f4446adc0a0: Acquire connection
2024-06-24 12:17:55.605,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [connection-manager] - id=0x7f4446adc0a0: Grabbing pooled connection (0x7f443de63f50)
2024-06-24 12:17:55.605,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=1, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=2, ref_count=1
2024-06-24 12:17:55.605,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: Successfully completed connection acquisition with connection id=0x7f443de63f50
2024-06-24 12:17:55.605,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-connection] - ConnManager Acquired Conn: conn: 0x7f443de63f50, manager: 0x7f4446adc0a0, err_code: 0,  err_str: Success.
2024-06-24 12:17:55.605,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-stream] - id=0x7f443de64c10: Created client request on connection=0x7f443de63f50: POST / HTTP/1.1
2024-06-24 12:17:55.614,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-stream] - id=0x7f443de64c10: Client request complete, response status: 200 (OK).
2024-06-24 12:17:55.614,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [http-connection] - ConnManager Releasing Conn: manager: 0x7f4446adc0a0, conn: 0x7f443de63f50
2024-06-24 12:17:55.614,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: User releasing connection (id=0x7f443de63f50)
2024-06-24 12:17:55.614,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ae4b38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=2, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=2, ref_count=1
2024-06-24 12:17:55.614,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [http-connection] - Requesting a new connection from conn_manager: 0x7f4446adc0a0
2024-06-24 12:17:55.614,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [connection-manager] - id=0x7f4446adc0a0: Acquire connection
2024-06-24 12:17:55.614,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [connection-manager] - id=0x7f4446adc0a0: Grabbing pooled connection (0x7f443de63f50)
2024-06-24 12:17:55.642,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=1, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=2, ref_count=1
2024-06-24 12:17:55.642,[DEBUG] [2024-06-24T12:17:55Z] [00007f444324eb38] [http-connection] - Requesting a new connection from conn_manager: 0x7f4446adc0a0
2024-06-24 12:17:55.642,[DEBUG] [2024-06-24T12:17:55Z] [00007f444324eb38] [connection-manager] - id=0x7f4446adc0a0: Acquire connection
2024-06-24 12:17:55.642,[DEBUG] [2024-06-24T12:17:55Z] [00007f444324eb38] [connection-manager] - id=0x7f4446adc0a0: Grabbing pooled connection (0x7f443d6e5c70)
2024-06-24 12:17:55.642,[DEBUG] [2024-06-24T12:17:55Z] [00007f444324eb38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=2, open_connection_count=2, ref_count=1
2024-06-24 12:17:55.642,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [connection-manager] - id=0x7f4446adc0a0: Successfully completed connection acquisition with connection id=0x7f443d6e5c70
2024-06-24 12:17:55.642,#
2024-06-24 12:17:55.642,# A fatal error has been detected by the Java Runtime Environment:
2024-06-24 12:17:55.642,#
2024-06-24 12:17:55.642,#  SIGSEGV (0xb) at pc=0x00007f4444dc6b55, pid=1, tid=47
2024-06-24 12:17:55.642,#
2024-06-24 12:17:55.643,# JRE version: OpenJDK Runtime Environment Corretto-11.0.23.9.1 (11.0.23+9) (build 11.0.23+9-LTS)
2024-06-24 12:17:55.643,# Java VM: OpenJDK 64-Bit Server VM Corretto-11.0.23.9.1 (11.0.23+9-LTS, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
2024-06-24 12:17:55.643,# Problematic frame:
2024-06-24 12:17:55.643,# C  [AWSCRT_2322636264284720268libaws-crt-jni.so+0x8bb55]  aws_java_http_stream_on_stream_destroy_fn+0x25
2024-06-24 12:17:55.643,#
2024-06-24 12:17:55.643,# Core dump will be written. Default location: /app/core.1
2024-06-24 12:17:55.643,#
2024-06-24 12:17:55.643,# An error report file with more information is saved as:
2024-06-24 12:17:55.643,# /app/hs_err_pid1.log
2024-06-24 12:17:55.649,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [http-connection] - ConnManager Acquired Conn: conn: 0x7f443d6e5c70, manager: 0x7f4446adc0a0, err_code: 0,  err_str: Success.
2024-06-24 12:17:55.649,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [http-stream] - id=0x7f443de65240: Created client request on connection=0x7f443d6e5c70: POST / HTTP/1.1
2024-06-24 12:17:55.649,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [http-stream] - id=0x7f443de65240: Client request complete, response status: 200 (OK).
2024-06-24 12:17:55.649,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [http-connection] - ConnManager Releasing Conn: manager: 0x7f4446adc0a0, conn: 0x7f443d6e5c70
2024-06-24 12:17:55.649,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [connection-manager] - id=0x7f4446adc0a0: User releasing connection (id=0x7f443d6e5c70)
2024-06-24 12:17:55.649,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=1, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=2, ref_count=1
2024-06-24 12:17:55.706,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [http-connection] - Requesting a new connection from conn_manager: 0x7f4446adc0a0
2024-06-24 12:17:55.706,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [connection-manager] - id=0x7f4446adc0a0: Acquire connection
2024-06-24 12:17:55.706,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [connection-manager] - id=0x7f4446adc0a0: Grabbing pooled connection (0x7f443d6e5c70)
2024-06-24 12:17:55.706,[DEBUG] [2024-06-24T12:17:55Z] [00007f4441accb38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=2, open_connection_count=2, ref_count=1
2024-06-24 12:17:55.706,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [connection-manager] - id=0x7f4446adc0a0: Successfully completed connection acquisition with connection id=0x7f443d6e5c70
2024-06-24 12:17:55.706,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [http-connection] - ConnManager Acquired Conn: conn: 0x7f443d6e5c70, manager: 0x7f4446adc0a0, err_code: 0,  err_str: Success.
2024-06-24 12:17:55.706,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [http-stream] - id=0x7f443de65bd0: Created client request on connection=0x7f443d6e5c70: POST / HTTP/1.1
2024-06-24 12:17:55.718,#
2024-06-24 12:17:55.718,# If you would like to submit a bug report, please visit:
2024-06-24 12:17:55.718,#   https://github.com/corretto/corretto-11/issues/
2024-06-24 12:17:55.718,#
2024-06-24 12:17:55.718,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [http-stream] - id=0x7f443de65bd0: Client request complete, response status: 200 (OK).
2024-06-24 12:17:55.718,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [http-connection] - ConnManager Releasing Conn: manager: 0x7f4446adc0a0, conn: 0x7f443d6e5c70
2024-06-24 12:17:55.718,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [connection-manager] - id=0x7f4446adc0a0: User releasing connection (id=0x7f443d6e5c70)
2024-06-24 12:17:55.718,[DEBUG] [2024-06-24T12:17:55Z] [00007f4444ac1b38] [connection-manager] - id=0x7f4446adc0a0: snapshot - state=1, idle_connection_count=1, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=2, ref_count=1

Reproduction Steps

Not reproducible deterministically right now

Possible Solution

No response

Additional Information/Context

Runs in a Fargate task for us

aws-crt-java version used

2.26.7 (happened for 2.21.29 as well)

Java version used

OpenJDK Runtime Environment Corretto-11.0.23.9.1

Operating System and version

amazoncorretto:11-alpine

### Tasks
TingDaoK commented 3 months ago

Hi, thanks for reporting the issue, but with the given info, we still cannot locate where the issue is. The debug level log shows everything was just working as expected before crash.

Firstly, if you can provide some steps to reproduce, that will be very helpful.

2024-06-24 12:17:55.643,# Core dump will be written. Default location: /app/core.1
2024-06-24 12:17:55.643,#
2024-06-24 12:17:55.643,# An error report file with more information is saved as:
2024-06-24 12:17:55.643,# /app/hs_err_pid1.log

Secondly, /app/core.1 and /app/hs_err_pid1.log Those two file will also be helpful for us to track the bug.

Lastly, if possible, can you enable the trace level log? Looks like you have the debug level log, and it doesn't have enough information about the lifetime that may cause the crash. However, it's very likely that the log will not be very helpful, but can provide a bit more information about what happened.

github-actions[bot] commented 3 months ago

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

ZAlex1988 commented 2 months ago

We have the same issue on a docker container. Did you ever find the solution for this? OpenJDK 11, eclipse-temurin:11-jre-alpine, CRT client version 2.25.31

andreasparteli commented 2 months ago

No, we didn't find a solution unfortunately. And since it was running in a Fargate task, access to the error log files wasn't really possible/feasible. So we ended up switching to Netty in the end.

jmklix commented 2 months ago

@ZAlex1988 can you provide more details about your docker container? A minimal reproducible example with a dockerfile and code snippet would help us track this bug down.

ZAlex1988 commented 2 months ago

Some details for the container where the failures were observed: eclipse-temurin:11-jre-alpine Application running on a wildfly server. I cannot provide a sample reproducible configuration, as the error happens seemingly randomly. Currently we have changed the container type, and updated the AWS SDK version (from 2.25.31 to 2.26.25). I will update if we see this error again. It's hard to reproduce.

floverfelt commented 2 months ago

Similar to @ZAlex1988 - running a Wildfly server using Alpine Linux, java 11. I have attached the err_pid.log file. It seems to occur in low memory situations when the application has been running for a few minutes and is under moderate load. dump_local.txt

ZAlex1988 commented 1 month ago

We have enabled the core dumps. The dump file is large, please see it at https://drive.google.com/file/d/1Z76Cx0zHElyi6XhD8Q7t1wBa3lHzYTAY/view?usp=sharing

andreasparteli commented 1 month ago

Potentially our issue was caused by a memory leak in another library. Could that be an explanation?

After replacing AWS CRT by Netty IO we found and fixed a memory leak. Since I was curious if missing memory could have led to the SIGSEGV, we switched back to AWS CRT and it's running fine for a week now.