Feature Request: parquet-protobuf

[jtjeferreira]

Any tips how to implement support for https://github.com/apache/parquet-mr/tree/master/parquet-protobuf?

[mjakubowski84]

Hi! Could you explain what you want to achieve?

[jtjeferreira]

Hi! I have some protobuf messages, that I would like to write to parquet files without having to map to intermediary classes nor derive schemas etc using the parquet4s typeclasses. I am also usigng akka-streams, but I think thats an orthogonal concern...

[mjakubowski84]

So, regarding writing, AFAIR this writer is used internally in every place. In order to write from Protobuf we need to replace it with Protobuf version. Additionally, we won't use schema resolver but a Protobuf Message.

If you want to add Protobuf support to Parquet4s - then it's fantastic. However, I would prefer not to add Protobuf dependencies to the existing artifacts. So I think that we can create a new module. With small changes to this class we could use a different internal writer and reuse it in protobuf module. Similarly, the same could happen in Akka Stream.

[jtjeferreira]

If you want to add Protobuf support to Parquet4s

I was evaluating parquet4s for a future project and if we end up using parquet this feature would be useful, and I will try to this.

However, I would prefer not to add Protobuf dependencies to the existing artifacts. So I think that we can create a new module

I was also thinking about that, like a parquet4s-protobuf that depends on parquet-protobuf. However do you think it would possible to make a change in parquet4s-core so that a user can pass any Writer (for example AvroParquetWriter )

[marcinaylien]

However do you think it would possible to make a change in parquet4s-core so that a user can pass any Writer

Yes, I think so, as long as backwards compatibility is kept.

[flipp5b]

Hi!

With small changes to this class we could use a different internal writer and reuse it in protobuf module.

It would be great. The ability to provide a custom writer could have a big value for those who prefer to bypass RowParquetRecord creation for performance reasons. In my case (schema with thousands of columns), this badly lowers GC pressure and boost overall write performance.

I suppose, as a next step, it's also possible to automatically derive a WriteSupport instance for case classes.

[jtjeferreira]

I was evaluating parquet4s for a future project and if we end up using parquet this feature would be useful, and I will try to this.

The future project was delayed, so I am not working on this...

[mjakubowski84]

Thanks to @flipp5b, Parquet4s v2.10.0 supports writing Protobuf, or any kind of data supported by the original Java Parquet library, to a single file. Examples show how this can be achieved.

I wonder if a similar improvement could be introduced to viaParquet, which has many more features.