ydb-platform / ydb

YDB is an open source Distributed SQL Database that combines high availability and scalability with strong consistency and ACID transactions
https://ydb.tech
Apache License 2.0
4.02k stars 587 forks source link

[Apache Arrow Flight] Support DoPut #126

Open 4ertus2 opened 2 years ago

4ertus2 commented 2 years ago

We have BulkUpsert method to insert big portions of data. https://ydb.tech/en/docs/reference/ydb-sdk/recipes/bulk_upsert/

We also want to support Apache Arrow Flight interface for this issue:

It looks like we should support direct DoPut call for simple ETL/ELT purposes. With ability to make it more complicated later (CommandStatementUpdate).

BulkUpsert supports Apache Arrow as data source format. So it's quite easy to pass data itself. There could be some troubles to implement GetFlightInfo for it and to make a stream.

So it's a good fist task in case of one portion insertion. And it has several extensions:

StepanovAndreyPsk commented 2 years ago

Currently working on this task in the issue KIKIMR-16464