Example for building an external index for parquet files

Is your feature request related to a problem or challenge?

It is common in databases and other analytic system to have additional external "indexes" (perhaps stored in the "metadata catalog", perhaps stored alongside the data files, perhaps embedded in the files, perhaps elsewhere)

These indexes are used to speed up queries by "pruning" the files -- specifically evaluating a predicate on the index and then only reading files / portions of files that would pass the filters in the query

Implementing such a index requires several three steps:

Creating / maintaining the actual index
A way to evaluate the index against the query predicate
A way to use the index to filter the files that need to be read

DataFusion has code already to do 2 (PruningPredicate) and 3 (ParquetExec) but I am not sure how obvious it is to put togeher

DataFusion actually also has basic support for 1 (e.g. the ListingTableProvider reads statistics and prunes based on their statistics)

We have some version of this in InfluxDB today, and we are in the process of extending it. However, I think the usecase is much more general and could be useful for other systems as well (e.g. full text indexes across parquet, for example).

I also think creating an example will help motivate the work from @NGA-TRAN in https://github.com/apache/datafusion/issues/10453 and myself in https://github.com/apache/datafusion/issues/9929

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

apache / datafusion