Open syucream opened 4 years ago
retry to implement Arrow record typed intermediate representation, once more! I think we can gradually switch to that by below steps:
prototyping for PoC: (various inputs) -> map's -> arrow -> map's -> json -> parquet
remove parquet writing side Go intermediates: (various inputs) -> map's -> arrow -> json -> parquet
remove input side Go intermediates: (various inputs) -> arrow -> json -> parquet
ideal: (various inputs) -> arrow -> parquet
parquet-go
retry to implement Arrow record typed intermediate representation, once more! I think we can gradually switch to that by below steps:
prototyping for PoC: (various inputs) -> map's -> arrow -> map's -> json -> parquet
remove parquet writing side Go intermediates: (various inputs) -> map's -> arrow -> json -> parquet
remove input side Go intermediates: (various inputs) -> arrow -> json -> parquet
ideal: (various inputs) -> arrow -> parquet
parquet-go
)