Container exited with a non-zero exit code 137 | Out of memory

[Vivek-Merugu]

I am encountering an issue while using the ABRiS library version 6.3.0 in my Spark Streaming Scala code.

• Spark version is 3.2.0,
• Scala version is 2.12.15,
• Kafka clients version is 3.2.0,
• io.confluent-kafka-avro-serializer version is 7.1.1,
• io.confluent-kafka-schema-registry-client version is 7.1.1,

• and io.confluent-kafka-schema-serializer version is 7.1.1.

  The ABRiS library has been instrumental in facilitating the De-Serialization/Serialization of Confluent AVRO messages in my Spark streaming application. Initially, everything works seamlessly for a few minutes. However, the Spark job eventually fails, and I receive the following exception:

  ERROR Client: Application diagnostics message: Application application_1696171462852_0173 failed 2 times due to AM Container for appattempt_1696171462852_0173_000002 exited with exitCode: 137
  Failing this attempt.Diagnostics: Container killed on request. Exit code is 137
  Container exited with a non-zero exit code 137.
  Killed by external signal

I have attempted to address the issue by increasing the driver memory and executor memory, but unfortunately, the problem persists.

Any guidance or assistance you can provide in resolving this issue would be greatly appreciated.

[cerveada]

What makes you believe that this issue is caused by Abris?

[Vivek-Merugu]

Thank you for your response. I appreciate your assistance in addressing this issue. Here's a breakdown of the situation:

Old Build Overview:

• The initial Spark Streaming Scala code processed JSON messages from Kafka Topic A.
• Operations were performed on these messages, and the results were published to Kafka Topic B.
• Successfully deployed on an Amazon EMR cluster, running smoothly.

New Build Overview:

• Transitioned to receiving Confluent AVRO messages from Kafka Topic A in the same application.
• Utilized ABRiS library functions (from_avro() and to_avro()) to deserialize AVRO to JSON and vice versa.
• The rest of the code remained unchanged.
• Successfully deployed on an Amazon EMR cluster, initially working fine for the first 20 to 30 minutes.

Error in New Build:

After the initial working period, the streaming application encountered the following error:

ERROR Client: Application diagnostics message: Application application_1696171462852_0173 failed 2 times due to AM Container for appattempt_1696171462852_0173_000002 exited with exitCode: 137
Failing this attempt.Diagnostics: Container killed on request. Exit code is 137
Container exited with a non-zero exit code 137.
Killed by external signal

Investigation Steps:

• Referenced the AWS Knowledge Center article here for troubleshooting.
• Despite following the suggested steps, the issue persists.

Specific Queries:

• Seeking your input on the possibility of ABRiS library functions (from_avro() and to_avro()) causing the issue.
• Additionally, exploring other potential factors, such as message payload size, that might be contributing to the problem.

Your insights and suggestions on resolving this matter would be greatly appreciated.

[cerveada]

Abris doesn't really works with json, it converts from avro to spark DataFrame. So it could also be some issue with json.

Best would be to use some profile to see what is really happening in the memory.

One Idea I have is the registryConfig. Do your application uses avro with many different registryConfig maps? That could cause a memory leak since the registry clients are cached for each config.

[Vivek-Merugu]

ABRiS Library Usage:

• Yes, I am utilizing the ABRiS library function from_avro() to convert AVRO to Spark DataFrame and then transforming the Spark DataFrame to JSON messages and vice-versa.

RegistryConfig in Use:

• The application employs three distinct registryConfig maps.

Memory Monitoring:

• Certainly, I will implement memory monitoring and keep you updated.

[Vivek-Merugu]

The problem happened because there wasn't enough memory allocated to the Spark executor. After increasing the Spark executor memory, the issue was resolved.

Hence, closing this issue.