search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.23k stars 3.58k forks source link

Failed to create source - Failed to transmit entry: -6 #11191

Closed luoyi-yu closed 1 year ago

luoyi-yu commented 3 years ago

Describe the question The following error occurred when I created the source.

[root@node1 apache-pulsar-2.6.1]# pulsar-admin source create \ --archive connectors/pulsar-io-debezium-mysql-2.6.1.nar \ --classname org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource \ --name debezium-mysql-source \ --destination-topic-name debezium-mysql-topic \ --tenant public \ --namespace default \ --parallelism 1 \ --source-config-file monitor-mysql.yaml Failed to transmit entry : -6

Reason: HTTP 500 Internal Server Error

monitor-mysql.yaml

configs: database.hostname: "192.168.2.174" database.port: "3306" database.serverTimezone: "GMT+8" database.user: "root" database.password: "root123" database.server.id: "174" database.server.name: "dbserver1" database.whitelist: "test" table.whitelist: "test.t_user" database.history: "org.apache.pulsar.io.debezium.PulsarDatabaseHistory" database.history.pulsar.topic: "my-history-topic" database.history.pulsar.service.url: "pulsar://node1:6650,node2:6650,node3:6650" key.converter: "org.apache.kafka.connect.json.JsonConverter" value.converter: "org.apache.kafka.connect.json.JsonConverter" pulsar.service.url: "pulsar://node1:6650,node2:6650,node3:6650" offset.storage.topic: "offset-topic"

The error in logs/pulsar-broker-node1.log is as follows 11:05:33.454 [pulsar-web-44-5] ERROR org.apache.pulsar.functions.worker.rest.api.SourcesImpl - Failed process Source public/default/debezium-mysql-source package: org.apache.distributedlog.exceptions.BKTransmitException: Failed to transmit entry : -6 at org.apache.distributedlog.BKLogSegmentWriter.lambda$new$0(BKLogSegmentWriter.java:243) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0] at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952) ~[?:1.8.0_191] at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) ~[?:1.8.0_191] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_191] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[?:1.8.0_191] at org.apache.distributedlog.BKTransmitPacket.notifyTransmitComplete(BKTransmitPacket.java:55) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0] at org.apache.distributedlog.BKLogSegmentWriter.addComplete(BKLogSegmentWriter.java:1244) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0] at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:91) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:430) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1784) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.client.LedgerHandle$5.safeRun(LedgerHandle.java:574) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191] Suppressed: org.apache.distributedlog.exceptions.WriteException: Write rejected because stream public/default/debezium-mysql-source/c5a19279-c9db-4b9f-82bc-7a03a64a0c38-pulsar-io-debezium-mysql-2.6.1.nar has encountered an error : writer has been closed due to error. at org.apache.distributedlog.BKAsyncLogWriter.doGetLogSegmentWriter(BKAsyncLogWriter.java:218) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0] at org.apache.distributedlog.BKAsyncLogWriter.getLogSegmentWriter(BKAsyncLogWriter.java:208) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0] at org.apache.distributedlog.BKAsyncLogWriter.getLogSegmentWriterForEndOfStream(BKAsyncLogWriter.java:237) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0] at org.apache.distributedlog.BKAsyncLogWriter.markEndOfStream(BKAsyncLogWriter.java:475) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0] at org.apache.distributedlog.AppendOnlyStreamWriter.markEndOfStream(AppendOnlyStreamWriter.java:82) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0] at org.apache.pulsar.functions.worker.dlog.DLOutputStream.close(DLOutputStream.java:69) ~[org.apache.pulsar-pulsar-functions-worker-2.6.1.jar:2.6.1] at org.apache.pulsar.functions.worker.WorkerUtils.uploadToBookeeper(WorkerUtils.java:95) ~[org.apache.pulsar-pulsar-functions-worker-2.6.1.jar:2.6.1]

It is worth mentioning that the related processes of pulsar seem to be running normally, the pulsar-client code can produce and consume data normally, and the following commands can also be displayed normally.

[root@node1 apache-pulsar-2.6.1]# pulsar-admin brokers list pulsar-cluster \ "node1:8080" "node2:8080" "node3:8080" [root@node1 apache-pulsar-2.6.1]# pulsar-admin source list --tenant public --namespace default \ [] [root@node1 apache-pulsar-2.6.1]# curl node2:8080/admin/v2/worker/cluster | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 250 100 250 0 0 3857 0 --:--:-- --:--:-- --:--:-- 3906 [ { "workerId": "c-pulsar-cluster-fw-node3-8080", "workerHostname": "node3", "port": 8080 }, { "workerId": "c-pulsar-cluster-fw-node2-8080", "workerHostname": "node2", "port": 8080 }, { "workerId": "c-pulsar-cluster-fw-node1-8080", "workerHostname": "node1", "port": 8080 } ]

The configuration information is as follows.

conf/bookkeeper.conf

advertisedAddress=node1 zkServers=node1:2181,node2:2181,node3:2181 journalDirectories=data/bookkeeper/journal ledgerDirectories=data/bookkeeper/ledgers prometheusStatsHttpPort=8100

conf/broker.conf

advertisedAddress=node1 clusterName=pulsar-cluster functionsWorkerEnabled=true zookeeperServers=node1:2181,node2:2181,node3:2181 configurationStoreServers=node1:2181,node2:2181,node3:2181

conf/functions_worker.yaml

pulsarFunctionsCluster: pulsar-cluster numFunctionPackageReplicas: 3

Can someone answer this question for me? please. Thanks in advance.

Operating System

codelipenghui commented 2 years ago

The issue had no activity for 30 days, mark with Stale label.

tisonkun commented 1 year ago

Closing as stale. If it's still relevant to maintained versions, feel free to open a new issue.

I appreciate the report is in details but we release a lot of more versions so there can be less interest for other community members to reproduce this issue in maintained versions.