[epic] Write structured columnar data

[bkirwi]

Product outcome

Persist has the ability to generate Parquet files that split out the columns in Row to distinct columns at the Parquet level, which comes with a host of benefits... but this has never been enabled in production.

Enabling this format will have immediate throughput and storage-usage benefits, and will unlock longer-term projects like schema migration and projection pushdown.

Discovery

This has been prototyped in main but never durably stored in S3.

Work items

Decision log

• 2024-01-30 Issue created.

[ParkMyCar]

@danhhz and I chatted today about how to order the work:

1. Get existing PRs reviewed and merged that write structured data to S3 and validate the structured data matches the codec data. Have the following feature flag/configuration options:
   • Everything off, read and write Codec data
   • Write Codec + Structured data
   • Decode Codec + Structured data and validate they match, if structured data is available
   • Decode only structured data, if its available
2. Iterate on the structured data until nightly is green, specifically we know there will be issues with persist-txn that are fairly complex.
3. Once nightlies are green, roll the feature flag out to staging, and monitor three things.
   • No panics
   • CPU usage
   • Memory usage
4. If staging looks good, then roll to production canaries, specifically focusing on the Analytics environment.
5. If staging and prod canaries look good with reading and validating structured data, start to serve reads from just the structured data.
6. Roll reading and validating structured data to Production environments, checking with Field Engineering for which environments we should start with.

While these changes are baking we can work on two things:

1. Projection pushdown on the k, v, k_s, and v_s columns. A concern we have is writing Codec and structured data is it will increase the size of blobs. We can alleviate this concern by fetching only the data we need.
2. V1 Arrow implementations of all the Datums we currently encode as protobuf bytes.

[ParkMyCar]

Adding the adapter label just so it shows up on our teams board