VM error: Internal error: VM crashed

[millin]

Hello @morazow!

Sometimes I get the following error, but I can't find the reason. Cound you help? Maybe you know possible reason of such crash.

W-UDF-CL-SL-JAVA-1075: Skipping init, because init method cannot be found.
[main] INFO com.exasol.cloudetl.kafka.KafkaTopicDataImporter$ - Starting Kafka consumer for partition '0' at next offset '906128505' for node '0' and vm '140604604606536'.
[main] INFO org.apache.kafka.clients.consumer.ConsumerConfig - ConsumerConfig values:
    allow.auto.create.topics = true
    auto.commit.interval.ms = 5000
    auto.offset.reset = earliest
    bootstrap.servers = [kafka-01:9092, kafka-02:9092]
    check.crcs = true
    client.dns.lookup = use_all_dns_ips
    client.id = consumer-ExaDWH-1
    client.rack =
    connections.max.idle.ms = 540000
    default.api.timeout.ms = 60000
    enable.auto.commit = false
    exclude.internal.topics = true
    fetch.max.bytes = 209715200
    fetch.max.wait.ms = 500
    fetch.min.bytes = 1
    group.id = ExaDWH
    group.instance.id = null
    heartbeat.interval.ms = 3000
    interceptor.classes = []
    internal.leave.group.on.close = true
    internal.throw.on.fetch.stable.offset.unsupported = false
    isolation.level = read_uncommitted
    key.deserializer = class com.exasol.cloudetl.kafka.deserialization.IgnoreKeyDeserializer$
    max.partition.fetch.bytes = 1048576
    max.poll.interval.ms = 300000
    max.poll.records = 500
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = [hidden]
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = PLAIN
    security.protocol = SASL_SSL
    security.providers = null
    send.buffer.bytes = 131072
    session.timeout.ms = 10000
    socket.connection.setup.timeout.max.ms = 30000
    socket.connection.setup.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
    ssl.endpoint.identification.algorithm = https
    ssl.engine.factory.class = null
    ssl.key.password = [hidden]
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.certificate.chain = null
    ssl.keystore.key = null
    ssl.keystore.location = /buckets/bfsdefault/kafka/ca.jks
    ssl.keystore.password = [hidden]
    ssl.keystore.type = JKS
    ssl.protocol = TLSv1.3
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.certificates = null
    ssl.truststore.location = /buckets/bfsdefault/kafka/ca.jks
    ssl.truststore.password = [hidden]
    ssl.truststore.type = JKS
    value.deserializer = class com.exasol.cloudetl.kafka.deserialization.AsStringDeserializer

[main] INFO org.apache.kafka.common.security.authenticator.AbstractLogin - Successfully logged in.
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 6.2.0-ccs
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 1a5755cf9401c84f
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1634206621469
[main] INFO org.apache.kafka.clients.consumer.KafkaConsumer - [Consumer clientId=consumer-ExaDWH-1, groupId=ExaDWH] Subscribed to partition(s): order_attributes-0
[main] INFO org.apache.kafka.clients.consumer.KafkaConsumer - [Consumer clientId=consumer-ExaDWH-1, groupId=ExaDWH] Seeking to offset 906128505 for partition order_attributes-0
[main] INFO org.apache.kafka.clients.Metadata - [Consumer clientId=consumer-ExaDWH-1, groupId=ExaDWH] Cluster ID: aVhFD8kQRFevL6md4ocEbQ
[main] INFO com.exasol.cloudetl.kafka.consumer.KafkaRecordConsumer - The last record offset for partition '0' is '906188812'.
[main] INFO com.exasol.cloudetl.kafka.consumer.KafkaRecordConsumer - Polled '500' records, total '500' records for partition '0' in node '0' and vm '140604604606536'.
[main] INFO com.exasol.cloudetl.kafka.consumer.KafkaRecordConsumer - Polled '106' records, total '606' records for partition '0' in node '0' and vm '140604604606536'.
[main] INFO com.exasol.cloudetl.kafka.consumer.KafkaRecordConsumer - Polled '500' records, total '1106' records for partition '0' in node '0' and vm '140604604606536'.
[main] INFO com.exasol.cloudetl.kafka.consumer.KafkaRecordConsumer - Polled '37' records, total '1143' records for partition '0' in node '0' and vm '140604604606536'.
[main] INFO org.apache.kafka.common.metrics.Metrics - Metrics scheduler closed
[main] INFO org.apache.kafka.common.metrics.Metrics - Closing reporter org.apache.kafka.common.metrics.JmxReporter
[main] INFO org.apache.kafka.common.metrics.Metrics - Metrics reporters closed
[main] INFO org.apache.kafka.common.utils.AppInfoParser - App info kafka.consumer for consumer-ExaDWH-1 unregistered
F-UDF-CL-LIB-1000:exaudfclient aborting ... cannot access socket file /tmp/zmqvmcontainer_conn_9703630733253653609.
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f09c31b3a10, pid=1, tid=2
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.18.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.18.04, mixed mode, sharing, tiered, compressed oops, serial gc, linux-amd64)
# Problematic frame:
# C  [libc.so.6+0x40a10]  abort+0x230
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log
#
# If you would like to submit a bug report, please visit:
#   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
#

[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f09c8426a10]

[morazow]

Hello @millin,

I do not know exactly. I am assuming maybe it is because of the memory? Usually it is the first thing we check. Could you please also provide SQL session logs? Then we could try to analyze and see if the memory limit reached.

[millin]

Hello @morazow, It seems you are right and it really happened due to lack of memory.

Thanks