search

awslabs / amazon-kinesis-producer

Amazon Kinesis Producer Library
Apache License 2.0
400 stars 331 forks source link

KPL intermittently, but infrequently, logs a code 56 curl error #126

Open behrangsa opened 7 years ago

behrangsa commented 7 years ago

Hi,

KPL (version 0.12.3) intermittently logs an error message like this:

[2017-08-31 20:18:34,810] 
ERROR [2017-08-31 20:18:34.810307] [0x000051f9][0x000070000051c000] [error]
[AWS Log: ERROR](CurlHttpClient)Curl returned error code 56
(com.amazonaws.services.kinesis.producer.LogInputStreamReader:64)

Looks like this error is not caught by the FutureCallback#onFailure method of the future returned by KinesisProducer#addUserRecord(...) though.

Thanks in advance.

tshearouse commented 7 years ago

I've been seeing this as well. I don't know how to reproduce it, but we see the same symptoms -- also not caught by the FutureCallback#onFailure method.

com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2017-08-17 19:06:06.503589] [0x00000013][0x00007f3f88826700] [error] AWS Log: ERRORCurl returned error code 56

We are calling the KPL from inside a Java Lambda function. I'd guess either there is a configuration error with the lambda container (is it clearing /tmp and nuking an ssl cert file?) or a network error (in our case, in us-east-1).

tshearouse commented 6 years ago

Following up because https://xkcd.com/979/

This curl error is not the cause of the lambda failures, but rather a poor log message that is a side effect of an unrelated timeout. The error appears to happen when a lambda times out and halts execution while attempting to place data onto a kinesis stream.

This error is a symptom of a performance problem elsewhere in your code. Either increase the lambda timeout, streamline your code so it will execute faster (or do less work inside the lambda), or both.

behrangsa commented 6 years ago

Off-topic, but IMHO, it would be better (I know, it is a subjective term), if the Java version of KCL was 100% pure Java and didn't depend on native compiled code. And Scala, Clojure, Kotlin could also produce pure Java classes so they were some other options too.

Now even debugging is much more involved, especially if I want to put breakpoints inside the C/C++ code.

nikhilkakade2108 commented 6 years ago

+1