search

databricks-demos / dbdemos

Demos to implement your Databricks Lakehouse
Other
255 stars 80 forks source link

llm-rag-chatbot has an issue on GCP as spark.conf.get("spark.databricks.clusterUsageTags.clusterWorkers") does not work #123

Open brianmcconnell opened 1 month ago

brianmcconnell commented 1 month ago

Just trying to run through the RAG LLM chatbot on GCP and the OCR loading is failing because:

spark.conf.get("spark.databricks.clusterUsageTags.clusterWorkers")

returns error

[[SQL_CONF_NOT_FOUND](https://docs.gcp.databricks.com/error-messages/error-classes.html#sql_conf_not_found)] The SQL config "spark.databricks.clusterUsageTags.clusterWorkers" cannot be found. Please verify that the config exists. SQLSTATE: 42K0I

This is not a demo issue per se but I think those tags are not guaranteed to exist.

brianmcconnell commented 1 month ago

None urgent as I just replaced it with a hard coded number from the cluster config