I ran the RAG demo about a week ago, and the vector search index was created in less than 30 minutes.
But today, I'm trying to run it again, and on the second attempt. Both of them got stuck in the Initializing step of the underlying DLT for more than 3 hours. Logs suggest it's waiting for some "capacity fulfillment" indefinitely, not sure what that is. Here's an excerpt from the logs, this message keeps showing for all the waiting time:
2024-03-03T11:15:06.010: Successfully created the index
*******************: 2024-03-03T11:15:06.124: batch: -1 Processing up to version 1: 17417 changes in 59 batches
*******************: 2024-03-03T11:15:06.302: batch: -1 / start waiting for capacity fulfillment, totalSourceTableNumRows: 17417
*******************: 2024-03-03T11:15:06.426: Waiting for capacity fulfillment before ingestion. pollingIntervalInMillis: 60000, maxWaitTime: 86400000, currentWaitTime: 60000
*******************: 2024-03-03T11:16:06.474: Waiting for capacity fulfillment before ingestion. pollingIntervalInMillis: 60000, maxWaitTime: 86400000, currentWaitTime: 120000
*******************: 2024-03-03T11:17:06.511: Waiting for capacity fulfillment before ingestion. pollingIntervalInMillis: 60000, maxWaitTime: 86400000, currentWaitTime: 180000
2024-03-03T11:17:42.930+0000: [GC (System.gc()) [PSYoungGen: 8342085K->74754K(15151616K)] 8871785K->604461K(46240256K), 0.0275154 secs] [Times: user=0.15 sys=0.00, real=0.03 secs]
I also get some warnings, not sure if related:
WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
WARN StatusConsoleListener RollingFileAppender 'publicFile.rolling': The bufferSize is set to 8192 but bufferedIO is not true
WARN StatusConsoleListener RollingFileAppender 'privateFile.rolling': The bufferSize is set to 8192 but bufferedIO is not true
WARN StatusConsoleListener RollingFileAppender 'com.databricks.UsageLogging.appender': The bufferSize is set to 8192 but bufferedIO is not true
WARN StatusConsoleListener RollingFileAppender 'com.databricks.EventLoggingStats.appender': The bufferSize is set to 8192 but bufferedIO is not true
WARN StatusConsoleListener RollingFileAppender 'com.databricks.ProductLogging.appender': The bufferSize is set to 8192 but bufferedIO is not true
WARN StatusConsoleListener RollingFileAppender 'com.databricks.LineageLogging.appender': The bufferSize is set to 8192 but bufferedIO is not true
WARN StatusConsoleListener RollingFileAppender 'com.databricks.MetricsLogging.appender': The bufferSize is set to 8192 but bufferedIO is not true
WARN StatusConsoleListener RollingFileAppender 'dltExecution.rolling': The bufferSize is set to 8192 but bufferedIO is not true
I ran the RAG demo about a week ago, and the vector search index was created in less than 30 minutes.
But today, I'm trying to run it again, and on the second attempt. Both of them got stuck in the Initializing step of the underlying DLT for more than 3 hours. Logs suggest it's waiting for some "capacity fulfillment" indefinitely, not sure what that is. Here's an excerpt from the logs, this message keeps showing for all the waiting time:
I also get some warnings, not sure if related: