[BUG] Incompatibility between Optuna 3.5.0 and Darts 0.27.1 due to Pytorch Lightning version conflict

[ivelin]

Describe the bug When using Darts with Optuna 3.5, there are errors due to the fact that Optuna switched to pip lightning and import lightning.pytorch while darts still uses pip pytorch-lightning and import pytorch_lightning

To Reproduce

1. Install latest Darts 0.27.2 and Optuna 3.5
2. Try the Darts with Optuna example: https://unit8co.github.io/darts/userguide/hyperparameter_optimization.html?highlight=gridsearch#hyperparameter-optimization-with-optuna
3. Or alternatively try this code derived from the example above: https://github.com/ivelin/canswim/blob/a032b99a37dcbec22cd208c175639e6d04837ba2/src/canswim/model.py#L704

Expected behavior Darts and Optuna should be able to coexist while both depending on Pytorch Lightning.

System (please complete the following information):

• Python version: [e.g. 3.11]
• darts version [e.g. 0.27.2]

Additional context I was able to work around the problem by downgrading optuna to 3.4.0 which is the version before they switched to pip lightning and import lightning.pytorch

[dennisbader]

Hi @ivelin, and thanks for raising this issue. It's true that we are still relying on pytorch-lightning rather than lightning.

I think it would be a good point for us to also start migrating to lightning. It shouldn;t be too much of an effort. I'll add a ticket for this and put it into the backlog.

As you said, for now I'd suggest to use optuna<=3.4.0.

[ivelin]

Thank you for clarifying, @dennisbader . That may help in the future with compatibility with other libraries in the torch ecosystem. Looks like the pytorch lightning team is steadily pushing in that direction.

[Borda]

It's true that we are still relying on pytorch-lightning rather than lightning.

Both packages shall have identical codebases for the same version, but it is true that inheritance and some internal checks may complain when a particular class is imported from an alternative package than Trainer...