Feature: support parquet files - Githubissues

streamingfast / substreams-sink-files

Binary application to consume your Substreams and output it's data out to files format (JSON, CSV, etc.)

Apache License 2.0

2 stars 4 forks source link

Feature: support parquet files #5

Open abourget opened 1 year ago

abourget commented 1 year ago

Would we need to have a schema definition attached to the program?

Features:

Dump parquet files in batch, with a batch size.
Provide a structured format for filenames. Example: {table_name}/{blocktime:%Y-%m-%d}/[start-blocknum]_[end-blocknum].parquet
The blocktime can be plucked from the Clock, and start and end blocknums being batch ranges.
Table name can be extracted from the objects.

Questions:

Would the proto model be DatabaseChanges? Would it expect only create operations?