Better Table config validation

[icefury71]

As mentioned in #5915 and #5899 we need a better way to validate table config. This includes following checks (not exhaustive):

• Validate column names : should match the ones in schema
• Time column check: name and type should be consistent with schema
• Table name sanity check (no space)

Indexing config validation

• Ensure that Inverted Index columns are not present in no dictionary columns
• Ensure that Bloom filter columns are not present in no dictionary columns

Retention and pushType

• Currently retention does not work if pushType is null (for real-time table). Check for a case where pushType is null and retention is configured for a table.

Tenant validation

• Ensure valid tenants are configured for server and broker

[npawar]

One more: https://github.com/apache/incubator-pinot/issues/5730

[npawar]

https://github.com/apache/incubator-pinot/issues/5900

[icefury71]

One more check: Currently retention does not work if pushType is null (for real-time table). Check for a case where pushType is null and retention is configured for a table.

[mcvsubbu]

With all the changes proposed/added, it will be useful if a utility is also provided to validate an existing tableconfig. Many of them are tightening the (lax) rules from before, and are valid. It will be unfortunate if an existing installation suddenly stopped working because code elsewhere (not just in the table addition path) starts assuming things about the tableconfig.

[kishoreg]

+1 to a utility to validate, we can add a validate endpoint in the controller

[icefury71]

Updated ticket description with all the suggestions.

[buchireddy]

Another use case: Validate that the bloomFilterColumns aren't BYTES type since that's not supported at this point.

cc @kishoreg @icefury71

[icefury71]

Another use case: Validate that the bloomFilterColumns aren't BYTES type since that's not supported at this point.

cc @kishoreg @icefury71

Looks like this should be supported, skipping validation for this particular thing

[buchireddy]

Another use case: Validate that the bloomFilterColumns aren't BYTES type since that's not supported at this point. cc @kishoreg @icefury71

Looks like this should be supported, skipping validation for this particular thing

@icefury71 Do you mean bloomFilterColumns supports BYTES columns? From which version/

[icefury71]

What I meant is - technically it should be supported (it's not today and we should probably fix that).

On Tue, Sep 29, 2020 at 12:31 AM Buchi Reddy Busi Reddy < notifications@github.com> wrote:

Another use case: Validate that the bloomFilterColumns aren't BYTES type since that's not supported at this point. cc @kishoreg https://github.com/kishoreg @icefury71 https://github.com/icefury71

Looks like this should be supported, skipping validation for this particular thing

@icefury71 https://github.com/icefury71 Do you mean bloomFilterColumns supports BYTES columns? From which version/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/incubator-pinot/issues/5942#issuecomment-700507663, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFRXV5JMUFTZ7U66D3KOFTSIGEN7ANCNFSM4QOTJTBA .

-- Thanks and regards

Chinmay Soman

[icefury71]

+1 to a utility to validate, we can add a validate endpoint in the controller

@kishoreg @mcvsubbu Seems like we already have a /tables/validate end-point in the Table API. This should be sufficient ? (Same thing with Schemas : /schemas/validate)

[mcvsubbu]

yes, that works as long as the same validations are performed. thanks

[icefury71]

yes, that works as long as the same validations are performed. thanks

Yes, its calling the same validation function underneath.

[Jackie-Jiang]

We can also add checks on partition config where the table can be partitioned on at most one column

[yupeng9]

+1

Another use case is the upcoming upsert. There are several expectations for upsert table like primary key in schema, replica group setup, routing by replicas group, partition config, ingestion via LLC etc