[Feature] add support pyarrow flight connector

trinodb / trino-python-client

Python client for Trino

Apache License 2.0

307 stars 150 forks source link

[Feature] add support pyarrow flight connector #416

Closed YuriyGavrilov closed 8 months ago

YuriyGavrilov commented 9 months ago

Describe the feature

just add the arrow flight RPC connector https://arrow.apache.org/docs/python/generated/pyarrow.flight.connect.html This is modern solution to ingest large amount of data.

Describe alternatives you've considered

No response

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

mosabua commented 9 months ago

What do you envision as the benefit of this?

If the python client connects to a flight server to get the data and runs on a client workstation and then submits the data to trino then the workstation is a bottleneck.

YuriyGavrilov commented 8 months ago

@mosabua It seems I was a little fascinated by the speed of technology and wrote down the idea in the wrong place :) Of course, you are right that sending data back to trino does not make much sense from a workstation and it is the server that must support the flight server, and Python already has enough libraries for connection. so close it.