Console output failed to be logged with offline WandbLogger

[JinchaoLove]

🐛 Bug

To Reproduce

https://colab.research.google.com/drive/1k-wWyw-_8P987-Ay7n-rtEZiGU2pIrev#scrollTo=zK7-Gg69kMnG

Expected behavior

The wandb will save the console outputs in the file wandb/latest-run/files/output.log, but no such file when using WandbLogger.

Environment

• Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
• PyTorch Lightning Version (e.g., 1.5.0): 1.7.1
• Lightning App Version (e.g., 0.5.2): N/A
• PyTorch Version (e.g., 1.10): 1.12.0+cu113
• Python version (e.g., 3.9): 3.7.13
• OS (e.g., Linux): Linux
• CUDA/cuDNN version: cu113
• GPU models and configuration: N/A
• How you installed PyTorch (conda, pip, source): pip
• If compiling from source, the output of torch.__config__.show(): N/A
• Running environment of LightningApp (e.g. local, cloud): local
• Any other relevant information: N/A

Additional context

Wandb related: https://github.com/wandb/wandb/issues/3978

cc @awaelchli @morganmcg1 @borisdayma @scottire @parambharat @manangoel99

[awaelchli]

@JinchaoLove You never passed the logger to the Trainer. You are missing

Trainer(
    ...
    logger=wblogger,
)

Once you do that, the logs will be written to the wandb folder.

[JinchaoLove]

Hi @awaelchli , thanks for the kind reply. I've updated the code in colab. It's expected to log all console outputs and be saved in wandb/lastest-run/files/output.log, but it's still missing. (It's in the files subfolder, not the logs subfolder). Below are the different logs using WandbLogger with pl.Trainer (bug: no output.log) and self-implemented WandbLogger with pl.lite.LightningLite (normally generated output.log).

• Using WandbLogger with pl.Trainer (bug: no output.log):

  

• Self-implemented WandbLogger with pl.lite.LightningLite (normally generated output.log):

  

[JinchaoLove]

• Why do we need output.log? Because sometimes useful information are printed in the console, such as the model summary, printed metrics, checkpoint/logs dirs, tracebacks, etc.

• How to mitigate current problem? The new release of wandb (v0.13.1) fixed the problem by using a server connect, but output.log only generated in online mode if we using WandbLogger with pl.Trainer. Please see in https://github.com/wandb/wandb/issues/3978#issuecomment-1212456758

[awaelchli]

@JinchaoLove Thanks. This is very strange, I can't tell exactly what could be the reason for this. I will have to compare what happens with using raw wandb api calls vs. using the WandbLogger wrapper.

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions - the Lightning Team!