Apache Arrow Flight SQL connector

[backkem]

Feature Request

Describe the feature you'd like: Hi all, I'm wondering if there would be interest in supporting an Apache Arrow Flight connector. This transport can enable faster data retrieval and higher throughput by reducing (de)serialization and data copying. It also gives you nice strict typing on query results. Further down the line it may also allow distributing of query processing without relaying the query results through a coordinator node.

Describe alternatives you've considered: None

Teachability, Documentation, Adoption, Migration Strategy: Apache Arrow Flight is a well documented protocol with implementations across many languages. Naturally, this would be an additive feature, next to the existing MySQL/ODBC Connector.

[winkyao]

@backkem hi, do you mean that TiDB supports Apache Arrow Flight as a new client to replace the MySQL protocol?

[winkyao]

And do you have any interest to develop this feature with us?

[backkem]

Hi @winkyao. Yes, but I would add it in addition to the MySQL protocol. The MySQL protocol has broad support in existing tooling, therefore I would not phase it out. However, if you're writing a new service and want increased performance, or any of the other benefits, you could switch over to the Apache Arrow Flight protocol. The Flight connector could also be more 'native' if the data is stored in the Apache Arrow format. I read it may already closely resemble it.

I'd love to help you build this but I'll have to find the time. Especially, to land it on my own. It may also be a good idea to reach out to the Arrow community as well, E.g.: I'm not sure if they have a standard for querying yet.

[winkyao]

@backkem Thanks for your suggestion. We will try to reach out to the Arrow community and find a way to cooperate with them.

[backkem]

I reached out and found out there is work being done on a Flight SQL Proposal:

• Proposal
• Slides
• Mail thread
• POC PR: ryannicholson/arrow#2

I'm sure some of the contributors here can provide valuable feedback on the design. Maybe we can experiment with an implementation as well.

[winkyao]

@zz-jason Could you please take a look at these designs?

[zz-jason]

@backkem Thank you for your suggestion!

In TiDB:

• We use Chunk, which is a columnar data structure like RecordBatch in Apache Arrow, to store data for vectorized query execution in the TiDB SQL layer. TiKV Coprocessor also utilizes a similar data structure and vectorized execution scheme.
• For internal communicating between TiDB and TiKV Coprocessor, we have used a similar method like Arrow Flight, which is named with chunk-rpc, you can found the proto here: SelectResponse
• For external communicating between the client and TiDB, we only support MySQL protocol at present.

After reading the blog about Arrow Flight and the proposal about Arrow Flight SQL, maybe we could:

• Try Arrow Flight for internal communication, which may help to:
  • improve the end to end query performance between TiDB and TiKV Coprocessor
  • enable TiDB SQL layer to query data stored in other external storage services
• Try Arrow Flight SQL for external communication (Client <-> TiDB), which enables other systems to query data stored in TiDB without MySQL protocol clients.

[zz-jason]

From the Implementation Status in Apache Arrow, seems we need to support Flight RPC for go firstly.

[backkem]

The first implementations of Flight SQL are shipping in Arrow 7.0.0: article. This doesn't include a Go port yet thought.

[backkem]

It looks like InfluxDB now has full support for Flight SQL: blog post.

[backkem]

Linking the Go Flight SQL package and server implementation example.

[backkem]

One major difference between the current MySQL connector and Arrow Flight is that the former is connection based and the latter uses a more stateless request/response design (gRPC).

Looking at the code, this may mean it makes more sense not the use the current session implementation and create a separate implementation that uses the Parser / Compiler / Executor directly. That being said, the RecordSet is already closely inspired by Apache Arrow.

[backkem]

Looking into it more, both the Compiler and Executor have a significant dependency on the sessionctx. This would either have to be unraveled or an ephemeral session could be created.

[backkem]

I created a very basic POC for this. You can find the code here:

• TiDB changes
• Usage example