search

databricks-demos / dbdemos

Demos to implement your Databricks Lakehouse
Other
255 stars 80 forks source link

dbdemos llm-rag-chatbot: DLT Pipeline failed after 'Waiting For Resources' #94

Closed ragarwal-ibm closed 5 months ago

ragarwal-ibm commented 5 months ago

Hi, I am getting the error where DLT pipeline gets stuck at 'Waiting For Resources'. This happens during creation and sync of vector search index in notebook '01-Data-Preparation-and-Index'. I can see the data in the table 'main.rag_chatbot.databricks_documentation'.

It is also giving below warning in DLT pipeline: Pinning your pipeline to a specified DBR version is not supported, as Delta Live Tables manages versions for all pipelines. Please remove this setting and instead set "channel": "CURRENT | PREVIEW" in your pipeline settings.

I do not find any option of edit this DLT pipeline (though I am admin for workspace, metastore and account).

image

I have workspace which is enabled for metastore. As of now, my metastore does not have any storage (ADLS path) associated, which is optional to have. But the catalog ('main') has the associated ADLS with all the permission in place and I can see table object at the specified storage location.

As per other post, I did execute the below command (as per other post) but it started giving other exceptions (which were not there before adding this command), so I reverted back. %pip install --ignore-installed Jinja2==3.1.2 markupsafe==2.1.1

Could you please suggest, what needs to be done here. Thanks,

ragarwal-ibm commented 5 months ago

I got the solution/Fix for this issue. Though it is optional to set storage (root) location for unity catalog metastore but index creation was not getting completed until i set the storage location. Once I set the storage location and sync the vector search index then DLT pipeline worked.

Please refer below link to set storage location to metastore: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/create-metastore#add-storage