Failures in RealtimeQuickStart when indexes are changed

[gortiz]

RealtimeQuickStart seems to fail when indexes are changed (either created, updated or removed).

In order to reproduce it:

• Start RealtimeQuickStart (or RealtimeQuickStartWithMinion, both seem to use minion anyway)
• Edit the table githubEvents and either remove the included inverted index or add a new one.
• Reload all segments using the UI button
• Wait until the reload finishes.
• Check the indexes in the UI.

The behavior is not 100% consistent. Sometimes it is needed to retry.

I've been abled to replicate this with both master(866c796bd56cf846b654f29f024f3e610557b2c7) and with release release-0.12.1. When done in master, the following log is printed:

java.lang.IllegalStateException: Failed to find table config for table: githubEvents_OFFLINE
    at com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[guava-20.0.jar:?]
    at org.apache.pinot.plugin.minion.tasks.BaseTaskExecutor.getTableConfig(BaseTaskExecutor.java:51) ~[classes/:?]
    at org.apache.pinot.plugin.minion.tasks.realtimetoofflinesegments.RealtimeToOfflineSegmentsTaskExecutor.convert(RealtimeToOfflineSegmentsTaskExecutor.java:124) ~[classes/:?]
    at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:217) ~[classes/:?]
    at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:77) ~[classes/:?]
    at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.runInternal(TaskFactoryRegistry.java:157) [classes/:?]
    at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.run(TaskFactoryRegistry.java:118) [classes/:?]
    at org.apache.helix.task.TaskRunner.run(TaskRunner.java:75) [helix-core-1.0.4.jar:1.0.4]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) [?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java) [?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:829) [?:?]

This log is not printed in 0.12.1.

Before change:

After inverted index is removed:

Also all older segments get deleted and new ones are created. The backup of the old ones are still indexed, while the new ones are not.

[gortiz]

I also tried to remove/create an inverted index in airlineStats, specifically in the Origin column, and it doesn't seem to break anything, so maybe the problem is only related to the github events table.

[Jackie-Jiang]

Per the exception message, I don't think it is related to the index change. The exception is thrown from RealtimeToOfflineSegmentsTask, which expects both REALTIME and OFFLINE table to exist. We should just remove this task from the table config

[shounakmk219]

I had a look at this, from the description it looks like there are 2 issues

1. One which @Jackie-Jiang pointed out that the exception log is due to the RealtimeToOfflineSegmentsTask task which is unable to find the respective offline table.

• If you see the QuickStartBase.DEFAULT_STREAM_TABLE_DIRECTORIES the githubEvents table entry alone is under examples/minions/stream while other tables are sitting under examples/stream.
• Even though there is a examples/stream/githubEvents but it does not follow the file name convention such as examples/stream/<TABLE_NAME>/<TABLE_NAME>_realtime_table_config.json and also has a different schema compared to the current githubEvents table.
• As examples/minions/stream/githubEvents is referenced at other places too so removing the task from the table config may not be the right way to handle this
• We can rename the current examples/stream/githubEvents to examples/stream/pullRequestMergedEvents in accordance to the naming convention and create a new examples/stream/githubEvents which is a copy of examples/minions/stream/githubEvents without the task in table config.

2. The one which @gortiz is pointing to where segments being deleted and index are acting weird upon index update + segment reload.

• The data in githubEvents table is from 2021 and the table config has retention set to 1 year so reload is deleting all the segments
• As there is no data the newly created consuming segment is showing the weird index view
• Increasing the retention time addresses the issue. Let me know if this makes sense or we should update the data? Both cases will require maintenance in future

[gortiz]

The data in githubEvents table is from 2021 and the table config has retention set to 1 year so reload is deleting all the segments

Very good catch! That would explain why we only see this behavior in this test and not in production and honestly it will make me sleep better :)