Redundant work when using both `io::write::FileWriter` and `io::flight::serialize_*` methods.

[GPSnoopy]

With my limited understanding (correct me if I'm wrong), apart from their different headers, I'd say that both the Feather file format and the Flight FlightData messages are identical.

In my application, I'm creating a Feather file and the corresponding FlightData messages, both using IPC Zstd compression.

If it is true that the data content is identical, it seems rather wasteful to compress twice the same data. Is there a way to write a Feather chunk from the already-compressed corresponding FlightData messages? If not, could we introduce such an API?

[jorgecarleitao]

Hey @GPSnoopy ! Yes, it is not possible. Yes it should be possible. :)

I think we would need something like:

• expose encode_chunk so that the user can use it instead of calling serialize_batch.
• add FileWriter::mut_dictionaries
• add FileWriter::write_encoded that FileWriter::write would use, but users could also use to write an already encoded chunk.