apache / arrow-cookbook

Apache Arrow Cookbook
https://arrow.apache.org/
Apache License 2.0
95 stars 46 forks source link

[PYTHON] Add a recipe for sending mixed schema data over a single Flight stream #353

Open nph opened 3 months ago

nph commented 3 months ago

Recently there was a question on the Arrow Users mailing list about how to send tables or record batches with multiple schemas over a single Flight stream. @lidavidm provided a helpful answer and mentioned that this question had come up in discussion previously. I thought the suggested solution would be a good addition to the cookbook since the idea of using a union of struct types may not be entirely obvious to newcomers to Flight and the problem itself is one that multiple people have run into (including me :-)

Anyway here's my attempt at a recipe. Any feedback gratefully appreciated.

lidavidm commented 3 months ago

This is perhaps not really Flight specific but it works here.

There's a separate PR to upgrade the PyArrow version, @amoeba should that go in first?