Allow dynamic setup in `setup()`

[gianmarcoaversanoenx]

🚀 Feature

Expose dataloaders to the LightningModule's setup method.

Motivation

This will allow for a real dynamic setup, meaning that some layers' size can be set up correctly thanks to access to the data. One does not know how data is coming from the trainer (via datamodule or dataloaders, or even from the model itself), so this must be done in setup in the most general way.

Pitch

def setup(self, stage: str = None) -> None:
        """Called at the beginning of fit (train + validate), validate, test, or predict. This is a good hook when you need to build models dynamically or adjust something about them. This hook is called on every process when using DDP.
        Args:
            stage (Optional[str]): either 'fit', 'validate', 'test', or 'predict'
        """
        if stage == "fit" and self.model.num_nodes_pmf is None:
            loaders: DataLoader = self.trainer.train_dataloader.loaders  # not possible at the moment
            # get cool info from the data: batch size, statistics, etc

Alternatives

At the moment, this is only possible via hooks:

def on_train_start(self) -> None:
        """Called at the beginning of training after sanity check."""
        if self.model.num_nodes_pmf is None:
            loaders: DataLoader = self.trainer.train_dataloader.loaders  # succeeds
            # do stuff

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Lite: enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

• Bolts: Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

• Lightning Transformers: Flexible interface for high-performance research using SOTA Transformers leveraging PyTorch Lightning, Transformers, and Hydra.

cc @borda

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions - the Lightning Team!

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions - the Lightning Team!

[awaelchli]

Thanks for the suggestion @gianmarcoaversanoenx

Setting the dataloader up so early could be difficult and possibly lead to breaking changes. I think a better approach for most users would be to pass the data-specific parameters as input arguments to the module. Then these can be accessed as attributes anywhere, including in setup even if the dataloaders haven't been enabled yet.