search

go-mysql-org / go-mysql

a powerful mysql toolset with Go
MIT License
4.58k stars 976 forks source link

Streaming MySql ResultSet #552

Open sananguliyev opened 3 years ago

sananguliyev commented 3 years ago

Hi,

I wanted to use MySQL proxy for big data analytics which sometimes takes a couple of hours to finish the execution have a full result. Is it possible to stream the ResultSet while the query is executing?

atercattus commented 3 years ago

Hello, @SananGuliyev.

There is Conn.ExecuteSelectStreaming which call callback for every row instead of saving the whole Resultset in memory. https://github.com/go-mysql-org/go-mysql/blob/master/README.md#example-for-select-streaming-v111

Maybe this is what you want?

sananguliyev commented 3 years ago

@atercattus it's what I need. I will test it. I do not know how I missed that. Thanks a lot.

EDIT: I have tested it but I could not do what I want to implement. I am currently using MySQL Server as a proxy and this method (https://github.com/go-mysql-org/go-mysql/blob/master/server/command.go#L16) should return the whole resultset.

atercattus commented 3 years ago

Got it. Then the functionality requires improvement from my side. We use streaming with "repacking" to another protocol :)

I'll make PR for you in the near future.

sananguliyev commented 3 years ago

Thanks a lot. Looking forward to your PR.

sananguliyev commented 2 years ago

Hi @atercattus is there any plan regarding to that enhancement? I could contribute in case you share your ideas how you plan to implement it.

atercattus commented 2 years ago

I changed my work and didn't do this task when I was going to... Thanks for your patience and this reminder. I will think over the issue before the end of the year. it's ok?

sananguliyev commented 2 years ago

Yes, sure. It's totally ok. Thanks for update.

sananguliyev commented 2 years ago

Hi @atercattus,

is there any update?

skoef commented 2 years ago

hey @sananguliyev, I recently worked on the streaming results from SELECT queries, since I had trouble implementing the initial design as well. Perhaps my refactor works better for you? #596

sananguliyev commented 1 year ago

Thankd @skoef, it looks what I am looking for. I will check it.