org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException

adrianovieira commented 3 years ago

This error occurs after importing some data to elasticsearch

environment

OS: Linux Fedora-33
elasticsearch: 7.11.2
ssl: yes; self-signed certificate
java: 
openjdk version "1.8.0_265"
OpenJDK Runtime Environment (Zulu 8.48.0.53-CA-linux64) (build 1.8.0_265-b11)
OpenJDK 64-Bit Server VM (Zulu 8.48.0.53-CA-linux64) (build 25.265-b11, mixed mode)

Embulk and plugins

Started Embulk v0.9.23
Loaded plugin embulk-input-postgresql (0.11.1)
Loaded plugin embulk-output-elasticsearch (0.4.7)
Loaded plugin embulk-filter-column (0.7.1)

logging:

2021-03-17 16:08:24.241 -0300 [INFO] (0025:task-0000): SQL: FETCH FORWARD 10000 FROM cur
2021-03-17 16:08:37.086 -0300 [INFO] (0025:task-0000): > 12,85 seconds
2021-03-17 16:08:37.088 -0300 [INFO] (0025:task-0000): Fetched 500 rows.
2021-03-17 16:08:37.089 -0300 [INFO] (0025:task-0000): Fetched 1.000 rows.
2021-03-17 16:08:37.090 -0300 [INFO] (0025:task-0000): Fetched 2.000 rows.
2021-03-17 16:08:37.093 -0300 [INFO] (0025:task-0000): Fetched 4.000 rows.
2021-03-17 16:08:37.096 -0300 [INFO] (0025:task-0000): Fetched 8.000 rows.
2021-03-17 16:08:37.098 -0300 [INFO] (0025:task-0000): SQL: FETCH FORWARD 10000 FROM cur
2021-03-17 16:08:37.191 -0300 [INFO] (0025:task-0000): > 0,09 seconds
2021-03-17 16:08:37.192 -0300 [INFO] (0025:task-0000): Fetched 500 rows.
2021-03-17 16:08:37.192 -0300 [INFO] (0025:task-0000): Fetched 1.000 rows.
2021-03-17 16:08:37.193 -0300 [INFO] (0025:task-0000): Fetched 2.000 rows.
2021-03-17 16:08:37.195 -0300 [INFO] (0025:task-0000): Fetched 4.000 rows.
2021-03-17 16:08:37.197 -0300 [INFO] (0025:task-0000): Fetched 8.000 rows.
2021-03-17 16:08:37.199 -0300 [INFO] (0025:task-0000): SQL: FETCH FORWARD 10000 FROM cur
2021-03-17 16:08:37.255 -0300 [INFO] (0025:task-0000): > 0,06 seconds
2021-03-17 16:08:37.256 -0300 [INFO] (0025:task-0000): Fetched 500 rows.
2021-03-17 16:08:37.256 -0300 [INFO] (0025:task-0000): Fetched 1.000 rows.
2021-03-17 16:08:37.257 -0300 [INFO] (0025:task-0000): Fetched 2.000 rows.
2021-03-17 16:08:37.259 -0300 [INFO] (0025:task-0000): Fetched 4.000 rows.
2021-03-17 16:08:37.260 -0300 [INFO] (0025:task-0000): SQL: FETCH FORWARD 10000 FROM cur
2021-03-17 16:08:37.261 -0300 [INFO] (0025:task-0000): > 0,00 seconds
2021-03-17 16:08:37.263 -0300 [INFO] (0001:transaction): {done:  1 / 1, running: 0}
org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]
    at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:340)
    at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:566)
    at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
    at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
    at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
    at org.embulk.spi.Exec.doWith(Exec.java:22)
    at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
    at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:242)
    at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:291)
    at org.embulk.EmbulkRunner.run(EmbulkRunner.java:155)
    at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
    at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
    at org.embulk.cli.Main.main(Main.java:64)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]
    at com.google.common.base.Throwables.propagate(Throwables.java:160)
    at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper.requestWithRetry(Jetty92RetryHelper.java:164)
    at org.embulk.output.elasticsearch.ElasticsearchHttpClient.sendRequest(ElasticsearchHttpClient.java:322)
    at org.embulk.output.elasticsearch.ElasticsearchHttpClient.push(ElasticsearchHttpClient.java:84)
    at org.embulk.output.elasticsearch.ElasticsearchRecordBuffer.bufferRecord(ElasticsearchRecordBuffer.java:68)
    at org.embulk.base.restclient.RestClientPageOutput.add(RestClientPageOutput.java:44)
    at org.embulk.spi.PageBuilder.doFlush(PageBuilder.java:214)
    at org.embulk.spi.PageBuilder.flush(PageBuilder.java:219)
    at org.embulk.spi.PageBuilder.addRecord(PageBuilder.java:198)
    at org.embulk.filter.column.ColumnFilterPlugin$1.add(ColumnFilterPlugin.java:237)
    at org.embulk.exec.LocalExecutorPlugin$ScatterTransactionalPageOutput$OutputWorker.call(LocalExecutorPlugin.java:351)
    at org.embulk.exec.LocalExecutorPlugin$ScatterTransactionalPageOutput$OutputWorker.call(LocalExecutorPlugin.java:291)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.eclipse.jetty.io.EofException
        ... 16 more
    Caused by: java.util.concurrent.ExecutionException: org.eclipse.jetty.io.EofException
        at org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:228)
        at org.embulk.util.retryhelper.jetty92.StringJetty92ResponseEntityReader.getResponse(StringJetty92ResponseEntityReader.java:31)
        at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper$1.call(Jetty92RetryHelper.java:107)
        at org.embulk.spi.util.RetryExecutor.run(RetryExecutor.java:81)
        at org.embulk.spi.util.RetryExecutor.runInterruptible(RetryExecutor.java:62)
        at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper.requestWithRetry(Jetty92RetryHelper.java:95)
        ... 14 more
    Caused by: org.eclipse.jetty.io.EofException
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:765)
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)
        at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:364)
        at org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:97)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        ... 1 more
    Caused by: java.io.IOException: Pipe quebrado
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170)
        ... 7 more
    Suppressed: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.eclipse.jetty.io.EofException
        ... 16 more
    Caused by: java.util.concurrent.ExecutionException: org.eclipse.jetty.io.EofException
        at org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:228)
        at org.embulk.util.retryhelper.jetty92.StringJetty92ResponseEntityReader.getResponse(StringJetty92ResponseEntityReader.java:31)
        at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper$1.call(Jetty92RetryHelper.java:107)
        at org.embulk.spi.util.RetryExecutor.run(RetryExecutor.java:81)
        at org.embulk.spi.util.RetryExecutor.runInterruptible(RetryExecutor.java:62)
        at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper.requestWithRetry(Jetty92RetryHelper.java:95)
        ... 14 more
    Caused by: org.eclipse.jetty.io.EofException
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:765)
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)
        at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:364)
        at org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:97)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        ... 1 more
    Caused by: java.io.IOException: Pipe quebrado
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170)
        ... 7 more
Caused by: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]
    at org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:228)
    at org.embulk.util.retryhelper.jetty92.StringJetty92ResponseEntityReader.getResponse(StringJetty92ResponseEntityReader.java:31)
    at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper$1.call(Jetty92RetryHelper.java:107)
    at org.embulk.spi.util.RetryExecutor.run(RetryExecutor.java:81)
    at org.embulk.spi.util.RetryExecutor.runInterruptible(RetryExecutor.java:62)
    at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper.requestWithRetry(Jetty92RetryHelper.java:95)
    ... 14 more
Caused by: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:277)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1309)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:182)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:129)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69)
    at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:89)
    at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:122)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    ... 1 more

Error: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]

hiroyuki-sato commented 3 years ago

Hello, @adrianovieira

I'm not familiar with much Elasticsearch. Could you tell me the following?

Did you import your certificate into the JVM environment? (i.e use keytool.)
Have you ever tried import without SSL I would like to know whether this issue relates to SSL or not.

Best regards.

adrianovieira commented 3 years ago

Hello, @adrianovieira

I'm not familiar with much Elasticsearch. Could you tell me the following?

Did you import your certificate into the JVM environment? (i.e use keytool.)

yes, I did it (if I hadn't done, the issue would be a certificate related).

Have you ever tried import without SSL

yes, on a "single-node" lab cluster! but it doesn't work for the production cluster which has more than one node and is over SSL.

I would like to know whether this issue relates to SSL or not.

Best regards.

hiroyuki-sato commented 3 years ago

Hello, @adrianovieira Thank you for your reply.

I'll try to create the same environment. I'll appreciate it if you send me the reproduce configuration of the elasticsearch.
Multiple clusters without SSL worked fine. Correct?

embulk / embulk-output-elasticsearch

org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException #64