POC/Arrow evaluation - Githubissues

as we delve deeper into the EPIC #50 it becomes apparent that maybe dig0data tech like arrow can help. So, can we do this...

basic questions

[x] convert THS objects into a arrow/parquet dataset that can be worked on easily using just regular FileSystemLike storage (including local and S3)
[x] compare performance querying and process large task (eg hazard aggregation in THP
[x] enumerate the pros/cons
[x] is parquet the preferred serialisation format
Sub questions:
[x] can we use arrows in-memory features and/or IPC techniques to boost performance and minimise file IO Plasma?
[x] can we do partitioning in arrow (not just parquet) how does that work see
[x] can we easily reshape datasets to optimise for different use-cases (3rd party , internal heavy compute)
[x] can we use SQL-like querys also here SELECT ...