Can not checkpoint and log

[lcaquot94]

The documentations says that when using with Ray Client, you must disable checkpointing and logging for your Trainer by setting checkpoint_callback and logger to False. So how can we log and save model during training ?

[bparaj]

I have been doing this:

1. import TuneReportCheckpointCallback from ray_lightning

   from ray_lightning.tune import TuneReportCheckpointCallback

2. Disable checkpointing with "enable_checkpointing": False, in the pl Trainer's configuration

3. Initialize logger:

   tb_logger = pl_loggers.TensorBoardLogger(save_dir="/tmp/some-dir")

4. Initialize tuning strategy

   from ray_lightning import RayStrategy
   strategy = RayStrategy(num_workers=1, num_cpus_per_worker=1, use_gpu=True)

5. Initialize trainer:

   trainer = pl.Trainer(
       **trainer_config,
       callbacks=[TuneReportCheckpointCallback({"accuracy": "accuracy"}, on="epoch_end")],
       strategy=strategy,
       logger=tb_logger
   )