extension v0.2.1 on exasol v7.0.4 crashes VM

[stvacs]

I am trying to make the extension v0.2.1 work with an exasol v7.0.4. When running the import statement the client receives a short note that the VM has crashed. Please see the attached files with commands/results and server-side exception log. commands.txt exasol_exception.log

I asked my colleague @ilikutle to take a look at the error message and the setup together with me. He confirmed the correct setup of the udfs & target table and the installation of jars (as expected as this step is automated and version controlled). During developing the feature AS_JSON_DOC he never observed such an exception (development was done on exasol v7.0.3).

Are there any known issues with v7.0.4? Why is it the VM crashing? What do i have to do to make this work?

Thank you in advance

[morazow]

Hello @stvacs,

Thanks for the feedback!

We are going to look into it. Could you please tell me if you used Exasol docker installation?

From the logs, I cannot see any possible reason for the crash. Could you please check the database logs of recent sessions and SQL statements?

Look for path inside the database node:

ls /d02_data/exadb/log/process/

For example:

$ cat /d02_data/exadb/log/process/2020*_SqlSession_1685685963777638400

[stvacs]

Hello @morazow,

We are going to look into it. Could you please tell me if you used Exasol docker installation?

yes, this is a testing instance based on the image exasol/docker-db:7.0.4

From the logs, I cannot see any possible reason for the crash. Could you please check the database logs of recent sessions and SQL statements?

There is one SqlSession log file matching: 20201210_111649_SqlSession_1685685963777638400.0.txt

[morazow]

Morning @stvacs,

Unfortunately, I cannot reproduce your issue to check it further. I followed similar setup with docker, using latest extension I was able to import from data into tables. I tested it with both external SQL client and ExaPlus.

We do not have any known issues for 7.0.4 release. And the logs do not provide clear reason(s) for VM crash.

Couple suggestions from our side:

• Could you please try using SQL client (DbVis, DataGrip, etc)?
• Is the avro record huge? Please try with simpler records, maybe it hits memory threshold and gets killed.

Logs below

[root@n11 logs]# nc -k -l 127.0.0.1 3000 > exasol_exception.log 2>&1 &                                       
[1] 1997  
[root@n11 logs]# /usr/opt/EXASuite-7/EXASolution-7.0.4/bin/Console/exaplus -u sys -p <password> -c localhost:8888 -f query.sql
EXAplus 7.0.4 (c) EXASOL AG                    

Friday, December 11, 2020 8:47:30 AM CET       
Connected to database DB1 as user sys.
EXASolution 7.0.4 (c) EXASOL AG                    

EXA: ALTER SESSION SET SCRIPT_OUTPUT_ADDRESS='127.0.0.1:3000';

Rows affected: 0                               

EXA: IMPORT INTO KAFKA_SCHEMA.TOPIC_EXASOL_TEST...

Rows affected: 1                               

EXA: SELECT * FROM KAFKA_SCHEMA.TOPIC_EXASOL_TEST;
...

Netcat logs:

[main] WARN org.apache.kafka.clients.consumer.ConsumerConfig - The configuration 'schema.registry.url' was supplied but isn't a known config.
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 6.0.0-ccs
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 17b744c31e00868b
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1607672854879
[main] INFO org.apache.kafka.clients.consumer.KafkaConsumer - [Consumer clientId=consumer-exasol-consumers-1, groupId=exasol-consumers] Subscribed to partition(s): exasol-test-0
[main] INFO org.apache.kafka.clients.consumer.KafkaConsumer - [Consumer clientId=consumer-exasol-consumers-1, groupId=exasol-consumers] Seeking to offset 0 for partition exasol-test-0
[main] INFO org.apache.kafka.clients.Metadata - [Consumer clientId=consumer-exasol-consumers-1, groupId=exasol-consumers] Cluster ID: 4Br79TthS260VJmTrlbp0A
[main] INFO org.apache.kafka.clients.Metadata - [Consumer clientId=consumer-exasol-consumers-1, groupId=exasol-consumers] Cluster ID: 4Br79TthS260VJmTrlbp0A
[main] INFO org.apache.kafka.clients.Metadata - [Consumer clientId=consumer-exasol-consumers-1, groupId=exasol-consumers] Cluster ID: 4Br79TthS260VJmTrlbp0A
[main] INFO com.exasol.cloudetl.kafka.KafkaTopicDataImporter$ - Emitted total '1' records for partition '2' in node '0' and vm '140569602276184'.
[main] INFO com.exasol.cloudetl.kafka.KafkaTopicDataImporter$ - Emitted total '0' records for partition '0' in node '0' and vm '140569602276176'.
[main] INFO com.exasol.cloudetl.kafka.KafkaTopicDataImporter$ - Emitted total '0' records for partition '1' in node '0' and vm '140569602276192'.

Query SQL:

[root@n11 logs]# cat query.sql 

ALTER SESSION SET SCRIPT_OUTPUT_ADDRESS='127.0.0.1:3000';

IMPORT INTO KAFKA_SCHEMA.TOPIC_EXASOL_TEST
FROM SCRIPT KAFKA_EXTENSION.KAFKA_CONSUMER WITH
    BOOTSTRAP_SERVERS   = 'kafka01.internal:9092,kafka02.internal:9093,kafka03.internal:9094'
    GROUP_ID            = 'exasol-consumers'
    SCHEMA_REGISTRY_URL = 'http://schema-registry:8081'
    TABLE_NAME          = 'KAFKA_SCHEMA.TOPIC_EXASOL_TEST'
    TOPIC_NAME          = 'exasol-test';

SELECT * FROM KAFKA_SCHEMA.TOPIC_EXASOL_TEST;

[morazow]

Hello @stvacs,

Were you able to check the query using SQL client? Or smaller Avro record?

[stvacs]

Hello @morazow, Thanks for your messages and please excuse my delayed response.

Couple suggestions from our side:

• Could you please try using SQL client (DbVis, DataGrip, etc)? I did my first tests with the connector using dbeaver before minimizing the error scenario to exaplus as client and running in Docker container. The messages from the JDBC driver were similar, but most importantly truncated. How would a DB client facilitate an error analysis? Running in the docker container allows to put the output in the same environment as other log files e.g. /exa/logs

• Is the avro record huge? Please try with simpler records, maybe it hits memory threshold and gets killed. The VM crashes 3-10 seconds after calling the udf, even if there is no message waiting in the queue. Sending very large messages would be another test case but not relevant here.

However, I had opportunity to test the container instance with other resource definitions (memory limit increased from 3.2GiB to 5.2GiB) and then the connector ran through. With the memory limit of 3.2GiB, no "OOM kill" messages were reported by the container runtime. It would also have resulted in the termination of the entire container and not just the crash of a internal Java VM.

I hope this helps to narrow down the conditions under which the crash occurs.

Best regards, Stefan

[stvacs]

Is there information available how much memory to set aside for the Java VM and the UDF when taking the message size and parameters into account? Thank you in advance.

[morazow]

Hello @stvacs,

Thanks for the update! Good to hear that you were able to run the import with increased memory.

Is there information available how much memory to set aside for the Java VM and the UDF when taking the message size and parameters into account?

We have a hard limit of 4GB memory for the Kafka import UDF.

That is, when running the import from Kafka, it internally creates virtual machine processes per Kafka topic partition (for parallelism), the total sum of the all memory consumed by these processes should not exceed 4GB.

[morazow]

Hello @stvacs,

Do you still have any questions? Can we close this issue?

If there are other questions or issues please feel free to open new Github issue.