search

streamnative / pulsar-rs

Rust Client library for Apache Pulsar
Other
369 stars 121 forks source link

consume engine stop working when broker pod is restarted #180

Open wexgjduv opened 3 years ago

wexgjduv commented 3 years ago

Hi,

I have pulsar deployed in kubernetes, when broker pods were restarted, the consume engine stopped working and no messages were consumed.

Here is the log.

[2021-11-18T17:58:41Z DEBUG pulsar::connection] Connecting to pulsar://172.24.138.47:6650: 172.24.138.47:6650                                                                                                     
[2021-11-18T17:58:41Z INFO  pulsar::connection_manager] Connected n°6884530024263700671 to pulsar://172.24.138.47:6650 in 705ms                                                                                   
[2021-11-18T17:58:42Z ERROR pulsar::connection_manager] could not ping connection 5713950516537300917 to the server at pulsar://172.24.144.226:6650: timeout when sending ping to the Pulsar server               
[2021-11-18T17:58:42Z DEBUG pulsar::consumer] starting the consumer engine for topic persistent://public/default/test-partition-0                                                                   
[2021-11-18T17:58:42Z ERROR pulsar::consumer] Consumer: connection 5713950516537300917 is not valid: Io(Custom { kind: TimedOut, error: "timeout when sending ping to the Pulsar server" })                       
[2021-11-18T17:58:42Z DEBUG pulsar::consumer] reconnecting consumer for topic: persistent://public/default/app-debug                                                                                  
[2021-11-18T17:59:11Z ERROR pulsar::connection_manager] could not ping connection 1149394182775037463 to the server at pulsar://172.24.163.7:6650: timeout when sending ping to the Pulsar server                 
[2021-11-18T17:59:11Z ERROR pulsar::connection_manager] could not ping connection 4844692989062234706 to the server at pulsar://172.24.136.70:6650: timeout when sending ping to the Pulsar server                
[2021-11-18T17:59:11Z ERROR pulsar::consumer] Consumer: connection 4844692989062234706 is not valid: Io(Custom { kind: TimedOut, error: "timeout when sending ping to the Pulsar server" })                       
[2021-11-18T17:59:11Z DEBUG pulsar::consumer] reconnecting consumer for topic: persistent://public/default/app-lala                                                                                  
[2021-11-18T17:59:11Z ERROR pulsar::consumer] Consumer: connection 1149394182775037463 is not valid: Io(Custom { kind: TimedOut, error: "timeout when sending ping to the Pulsar server" })                       
[2021-11-18T17:59:11Z DEBUG pulsar::consumer] reconnecting consumer for topic: persistent://public/default/pulsar                                                                                       
[2021-11-18T17:59:41Z DEBUG pulsar::connection_manager] ConnectionManager::connect(BrokerAddress { url: Url { scheme: "pulsar", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("pulsar- 
[2021-11-18T17:59:41Z DEBUG pulsar::connection] Connecting to pulsar://pulsar-broker.pulsar:6650: 172.24.170.9:6650                                                                                               
[2021-11-18T17:59:41Z INFO  pulsar::connection_manager] Connected n°672426720355428389 to pulsar://pulsar-broker.pulsar:6650 in 6ms                                                                               
[2021-11-18T17:59:42Z ERROR pulsar::connection_manager] could not ping connection 5713950516537300917 to the server at pulsar://172.24.144.226:6650: timeout when sending ping to the Pulsar server               
[2021-11-18T17:59:42Z ERROR pulsar::consumer] Consumer: connection 5713950516537300917 is not valid: Io(Custom { kind: TimedOut, error: "timeout when sending ping to the Pulsar server" })                       
[2021-11-18T17:59:42Z DEBUG pulsar::consumer] reconnecting consumer for topic: persistent://public/default/app-partition-0                                                                           
[2021-11-18T18:00:11Z ERROR pulsar::connection_manager] could not ping connection 1149394182775037463 to the server at pulsar://172.24.163.7:6650: timeout when sending ping to the Pulsar server                 
[2021-11-18T18:00:11Z ERROR pulsar::connection_manager] could not ping connection 4844692989062234706 to the server at pulsar://172.24.136.70:6650: timeout when sending ping to the Pulsar server                
[2021-11-18T18:00:11Z ERROR pulsar::consumer] Consumer: connection 4844692989062234706 is not valid: Io(Custom { kind: TimedOut, error: "timeout when sending ping to the Pulsar server" })                       
[2021-11-18T18:00:11Z DEBUG pulsar::consumer] reconnecting consumer for topic: persistent://public/default/app-check                                                                               
[2021-11-18T18:00:11Z ERROR pulsar::consumer] Consumer: connection 1149394182775037463 is not valid: Io(Custom { kind: TimedOut, error: "timeout when sending ping to the Pulsar server" })                       
[2021-11-18T18:00:11Z DEBUG pulsar::consumer] reconnecting consumer for topic: persistent://public/default/app-lala
[2021-11-18T18:00:41Z DEBUG pulsar::connection_manager] ConnectionManager::connect(BrokerAddress { url: Url { scheme: "pulsar", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("pulsar- 
[2021-11-18T18:00:41Z DEBUG pulsar::connection] Connecting to pulsar://pulsar-broker.pulsar:6650: 172.24.148.112:6650                                                                                             
[2021-11-18T18:00:41Z INFO  pulsar::connection_manager] Connected n°8270173198854529008 to pulsar://pulsar-broker.pulsar:6650 in 4ms                                                                              
[2021-11-18T18:00:42Z ERROR pulsar::connection_manager] could not ping connection 5713950516537300917 to the server at pulsar://172.24.144.226:6650: timeout when sending ping to the Pulsar server               
[2021-11-18T18:01:11Z ERROR pulsar::connection_manager] could not ping connection 1149394182775037463 to the server at pulsar://172.24.163.7:6650: timeout when sending ping to the Pulsar server                 
[2021-11-18T18:01:11Z ERROR pulsar::connection_manager] could not ping connection 4844692989062234706 to the server at pulsar://172.24.136.70:6650: timeout when sending ping to the Pulsar server                
[2021-11-18T18:01:11Z ERROR pulsar::consumer] Consumer: connection 4844692989062234706 is not valid: Io(Custom { kind: TimedOut, error: "timeout when sending ping to the Pulsar server" })                       
[2021-11-18T18:01:11Z DEBUG pulsar::consumer] reconnecting consumer for topic: persistent://public/default/app-cd
[2021-11-18T18:01:11Z ERROR pulsar::consumer] Consumer: connection 1149394182775037463 is not valid: Io(Custom { kind: TimedOut, error: "timeout when sending ping to the Pulsar server" })                       
[2021-11-18T18:01:11Z DEBUG pulsar::consumer] reconnecting consumer for topic: persistent://public/default/app-l-partition-0
DonghunLouisLee commented 2 years ago

I wasn't able to reproduce this issue with the version 4.1.2. Was this issue handled?