crflynn / pbspark

protobuf pyspark conversion
MIT License
21 stars 5 forks source link

convert timestamp to datetime by default #16

Closed crflynn closed 2 years ago

crflynn commented 2 years ago

Previously we made changes to convert bytes type to spark bytes by default, the reason being that spark has a binary type that we can convert to so we should do it automatically. https://github.com/crflynn/pbspark/pull/11

We don't take the same approach for Timestamp, instead we force the user to enable it manually.

Since TimestampType exists, we should just convert automatically.