search

databricks-demos / dbdemos

Demos to implement your Databricks Lakehouse
Other
286 stars 96 forks source link

"use_current_cluster=True" not being carried to all the tasks in the jobs #127

Closed mechevarria closed 3 months ago

mechevarria commented 5 months ago

The option to use the same compute as the notebook user_current_cluster=True isn't carried to the task load_dbsql_and_ml_data in the Lakehouse for IoT demo. A 13.3 cluster tries to get created instead. This causes the job to fail in environments with classic compute that require the enhanced security plugin (no encryption in transit). The Lakehouse C360 workflow has the same issue. If the current cluster is used for every task, then there is no issue

Screenshot 2024-06-12 at 3 54 04 PM
QuentinAmbard commented 5 months ago

hey @mechevarria , yes it's sadly by design as we need for now a shared compute to load that. We're working on a broader update to better support serverless compute in our content and hopefully it should improve this. For now I'd recommend you run the notebook query using a warehouse for example, it should do the trick. It's a bit of a temporary workaround

QuentinAmbard commented 3 months ago

it should be better with the latest release!