DLT Dataset (Lending Tree)?

[Fallinggator]

I tried to use the full DLT demo, but ran into a few problems.

1. Had to manually alter the cluster it tried to create to use a cluster policy since required by my org. (Resolved, but would be nice if aspects like that were easily configurable with args)

2. Failed to load the lending tree dataset csv. I couldn't find it anywhere in the workspace samples or in the repo. I looked on Kaggle, but there are too many potential matches. No idea which of the lending tree data sets it is expecting.

[karthik25-gk]

Having the same issue with the dataset csv. Can I know where I can download from.

[QuentinAmbard]

ok I'll have a look, must be a bundle issue

[QuentinAmbard]

@Fallinggator I'm not sure I'm looking at the right place, are you saying you don't have this dataset in your workspace ?

spark.read.csv('/databricks-datasets/lending-club-loan-stats', header=True) ?

Can you confirm which demo you're loading and where it's failing exactly ? Thanks

[karthik25-gk]

@QuentinAmbard Ignore my issue. It’s been resolved. Thanks

[QuentinAmbard]

Ok I understand what happened, becuase you can't create the cluster, the init job did not start, and the data wasn't generated. I'll instead start the init job with the current cluster, it'll be faster & solve these kind of issue.

[QuentinAmbard]

Hi @Fallinggator , we released a new option that you can enable in the intallation: use_current_cluster = True

You should now have a better error and a suggestion for you to use this option if we can't install a new cluster:

This should solve your 2 issues !

Longer term it'll likely become the default option

Feel free to reopen or let us know if you're still facing some issue while using this new option!