Add support for a pqrs schema --arrow option

manojkarthick / pqrs

Command line tool for inspecting Parquet files

Apache License 2.0

296 stars 29 forks source link

Add support for a pqrs schema --arrow option #51

Closed wynnw closed 5 months ago

wynnw commented 5 months ago

This prints out a json-serialized arrow schema, as defined by the arrow project api's for doing parquet -> arrow schema conversion. I've been trying to figure out how to define schemas to use for the csv2parquet project and this was a nice simple way to do that. I had to upgrade the arrow dependency version and fix a few build errors around that.

manojkarthick commented 5 months ago

Hi @wynnw, thank you for the contribution - appreciate it! Any chance you could add an integration test for this feature?

wynnw commented 5 months ago

I added a small test patterned after the other json output tests and fixed another test that was failing with the arrow dependency upgrades. So hopefully that's good? The tests could definitely be much more involved but that's all I had time for right now.

manojkarthick commented 5 months ago

Looks great, thank you so much for contributing this feature!