blackbeam / rust-mysql-simple

Mysql client library implemented in rust.
Apache License 2.0
658 stars 144 forks source link

multistatement queries enter infinite busy loop upon network error #306

Closed fulara closed 2 years ago

fulara commented 2 years ago

I have encountered an issue with multi statement queries. In my case those queries have multiple inserts to a different tables formatted and send as a string request to database.

If during the execution of the multistatment query a network failure happens, such as target endpoint going down, all writers hang.

version of the rust-mysql-simple: latest master

My understanding is that this is root of the problem read_packet in handle_result_set from line src/conn/mod.rs:914 returns an error. handle_next -> query_result:147 matches an error and sets self.state to Errored self.conn.more_results_exists() will keep on returning true because nothing touches status_flags in this flow.

Callers generally expect the code to return OnBoundary or Done at some point. death loop is entered via ResultSet drop call.

fn drop(&mut self) {
        while self.next().is_some() {}
    }
fn next()  .. { 
...
Errored(err) => {
  self.handle_next();
  Some(Err(err))
}

In case of network failure let's say the target endpoint goes down, read_packet will be returning error forever, so this will spin forever.

I have made a very silly commit that adds logs to confirm that above is the issue here is the commit: https://github.com/fulara/rust-mysql-simple/commit/186f615f191c6451a207555dfbf8a116cd6717eb

Here are log statements ( from one thread):

Silly log statements ``` ThreadId(15) Loggin! I am still loopin ThreadId(15) Loggin! state: in ::iter() Errored(IoError { server disconnected }) ThreadId(15) Loggin! state: OnBoundary index 582637288, conn: Mut(Conn(ConnInner { ... } , last_command: 3, connected: true, has_results: false, local_infile_handler: None })) more result exist?! true ThreadId(15) Reading a packet.... ThreadId(15) Loggin! Entered err state. now: Errored(IoError { server disconnected }), index b4 incrementing 582637288. err is: IoError { server disconnected } ThreadId(15) Loggin! I am still loopin ThreadId(15) Loggin! state: in ::iter() Errored(IoError { server disconnected }) ThreadId(15) Loggin! state: OnBoundary index 582637289, conn: Mut(Conn(ConnInner { ... } , last_command: 3, connected: true, has_results: false, local_infile_handler: None })) more result exist?! true ThreadId(15) Reading a packet.... ThreadId(15) Loggin! Entered err state. now: Errored(IoError { server disconnected }), index b4 incrementing 582637289. err is: IoError { server disconnected } ThreadId(15) Loggin! I am still loopin ```

You can say this is a more involved case of this: #285 Whcih probably does not work in this case because the code bails out earlier.

blackbeam commented 2 years ago

Thanks for report. I'll look into it as soon as possible.

fulara commented 2 years ago

hey @blackbeam, thanks for response.

this hacky attempt: https://github.com/fulara/rust-mysql-simple/commit/67747ada366e460dd45ab7a108c228aec69fbb3b seems to fix the issue, but no clue how it fits into err handling flow.

blackbeam commented 2 years ago

@fulara, thanks!