search

google / riegeli

Riegeli/records is a file format for storing a sequence of string records, typically serialized protocol buffers.
Apache License 2.0
418 stars 53 forks source link

Any chance of a Java port? #15

Open davidmankin opened 3 years ago

davidmankin commented 3 years ago

This looks like a really neat as a way to serialize a large number of protobuffers. I see there's C++ and Python but at least my reader process is Spark based, so Java. Any chance there's a Java port on the way? Or at least JNI/Swig bindings and README on how to make it work from Java?

QrczakMK commented 3 years ago

There is a long term plan to add it, likely in the form of JNI bindings.

But I am afraid I cannot promise any time horizon.

mzhaom commented 2 years ago

I might be able to provide some help. Wonder whether @QrczakMK you have any high level plan in mind.

The tricky part w.r.t JNI binding is java protobuf implementation is not done through C++ implementation(unlike python protobuf), so the record type handling in Record{Reader,Writer} might need a little bit work.

mzhaom commented 2 years ago

WIP PR: https://github.com/google/riegeli/pull/21

QrczakMK commented 2 years ago

Thank you!

The protobuf difference will not matter. Riegeli/Python does not rely on how Python protobuf is implemented, it never passes parsed protos across languages, it passes only serialized protos, which are parsed/serialized on the Python side if needed. The same can be done in Riegeli/Java.

On Wed, 29 Dec 2021 at 04:18, Ming Zhao @.***> wrote:

I might be able to provide some help. Wonder whether @QrczakMK https://github.com/QrczakMK you have any high level plan in mind.

The tricky part w.r.t JNI binding is java protobuf implementation is not done through C++ implementation(unlike python protobuf), so the record type handling in Record{Reader,Writer} might need a little bit work.

— Reply to this email directly, view it on GitHub https://github.com/google/riegeli/issues/15#issuecomment-1002376720, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACAUJJA22IPD4JS6AHIA5PDUTJ4W7ANCNFSM42RUL33Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

-- Marcin Kowalczyk