search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.53k stars 1.3k forks source link

Impossible to use the virtual column generated by the timestamp index ($ts$DAY) in the startree index #10213

Open enzo-dechaene opened 1 year ago

enzo-dechaene commented 1 year ago

When I try to create my realtime table with a timestamp and a startree index, I get this error :

Invalid table config: my-table-name_REALTIME. Column Name $ts$DAY defined in StarTreeIndex Config must be a valid column defined in the schema

Currently the table config validation requires all the star-tree index to be explicitly configured within the schema.

s0nskar commented 1 year ago

@Jackie-Jiang I would like to work on this but I will need some guidance for this.

Should we even allow creating indexes on top of generated columns? This will create a dependency on indexes which afaik Pinot doesn't support currently.

Jackie-Jiang commented 1 year ago

We have plans to re-design the timestamp index because currently it is more or less just the syntactic sugar for derived column + expression override. In regular case, derived column should be defined within the schema so that it can be used within the star-tree, but it is not the case for timestamp index as of now. Add @gortiz to share more ideas