[Databricks] - Neuralforecast makes everything slow and freezes the notebook

[vinaybridge]

Hello Team,

Describe the bug

When we start model training on Databricks, everything hangs and suddenly memory consumption increases on the local machine.

• My primary observation is that it does not happen with statsforecast package.
• I observed that statsforecast does not use tqdm.notebook (fancy tqdm), and Neuralforcast uses it for verbose. I even disabled verbose=False, but the same issue persists.
• After training, if we close the notebook and reopen it, everything freezes. Can anyone look into it?

Thanks

[kdgutier]

Hey @vinaybridge,

Parallelization of NeuralForecast models using databricks is not yet a feature of our package.

I rather recommend you to use specialized GPU hardware. You can access it directly from the tutorial examples:

I don't know what is the outcome of parallelizing the execution of NeuralForecast in Databricks, we have never done it.

[vinaybridge]

Yes, I understand that you are correct, but I am not using it for Parallelization. Just using normal code. Not sure how I can use NeuralForecast with standard CPU setting on Databricks.

Basically, everything stuck because of that UI tqdm widget