Open GPSnoopy opened 1 year ago
Hey @GPSnoopy ! Yes, it is not possible. Yes it should be possible. :)
I think we would need something like:
encode_chunk
so that the user can use it instead of calling serialize_batch
. FileWriter::mut_dictionaries
FileWriter::write_encoded
that FileWriter::write
would use, but users could also use to write an already encoded chunk.
With my limited understanding (correct me if I'm wrong), apart from their different headers, I'd say that both the Feather file format and the Flight
FlightData
messages are identical.In my application, I'm creating a Feather file and the corresponding
FlightData
messages, both using IPC Zstd compression.If it is true that the data content is identical, it seems rather wasteful to compress twice the same data. Is there a way to write a Feather chunk from the already-compressed corresponding
FlightData
messages? If not, could we introduce such an API?