search

crflynn / pbspark

protobuf pyspark conversion
MIT License
21 stars 5 forks source link

Unable to avoid PicklingErrors with latest protoc version #45

Closed amichel closed 1 year ago

amichel commented 1 year ago

I'm using protoc 22.0 and protobuf 4.22. I was trying to follow your guide on "Avoiding PicklingErrors". Unfortunately it seems that protobuf registers all types in the global scope and no explicit import of generated module is needed any longer. I am getting the pickling error every time, no matter the module name of the message class.

crflynn commented 1 year ago

I just tested a PR with the latest protoc+protobuf tooling. The test suite passed, but if you have any additional details on your situation I can try to take a deeper look.

amichel commented 1 year ago

I managed to solve the by loading protoc generated class via SparkContext.addPyFile and then importing.

On Wed, Mar 8, 2023, 13:47 Flynn @.***> wrote:

I just tested a PR with the latest protoc+protobuf tooling. The test suite passed, but if you have any additional details on your situation I can try to take a deeper look.

— Reply to this email directly, view it on GitHub https://github.com/crflynn/pbspark/issues/45#issuecomment-1460691406, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM5BY444GQDW4P46ZSJKWDW3DH4BANCNFSM6AAAAAAVTFFFTA . You are receiving this because you authored the thread.Message ID: @.***>