[Q] Console output failed to be logged in offline mode

[JinchaoLove]

Hi, there, I'm using a WandbLogger (wandb=0.12.21) in pytorch lightning 1.6.5, and find that the output.log file is empty when set offline mode. The problem disappear when using online mode. I've tried setting os.environ["WANDB_CONSOLE"] = "auto", "redirect", or "wrap", but not help. Could you help me to deal with this problem?

[MBakirWB]

Hi @JinchaoLove , when running wandb offline, we save the metrics locally and associated run logs. However, the output.log, does not get save locally and requires an activate connection with our servers. There is a feature request in place for this and it is actively being review for implementation. We will update you when available.

[JinchaoLove]

Hi @JinchaoLove , when running wandb offline, we save the metrics locally and associated run logs. However, the output.log, does not get save locally and requires an activate connection with our servers. There is a feature request in place for this and it is actively being review for implementation. We will update you when available.

Thanks for that! It would be great if it's done. Currently I override the builtin print to deal with the problem. Hope this could help others before the update.

class Print(object):
    def __init__(self, file=None, sink=print):
        self.file = file
        self.sink = sink

    def __call__(self, *args, **kwargs):
        self.sink(*args, **kwargs)  # show in console
        if self.file:  # redirect to file
            if kwargs and 'file' in kwargs.keys():
                return
            with open(self.file, 'a') as f:
                self.sink(*args, file=f, **kwargs)

_print = builtins.print
log_file = os.path.join(log_dir, 'wandb', 'latest-run', 'files', 'output.log')
builtins.print = Print(file=log_file, sink=_print)

[exalate-issue-sync[bot]]

WandB Internal User commented: JinchaoLove commented:

Hi @JinchaoLove , when running wandb offline, we save the metrics locally and associated run logs. However, the output.log, does not get save locally and requires an activate connection with our servers. There is a feature request in place for this and it is actively being review for implementation. We will update you when available.

Thanks for that! It would be great if it's done. Currently I override the builtin print to deal with the problem. Hope this could help others before the update.

class Print(object):
    def __init__(self, file=None, sink=print):
        self.file = file
        self.sink = sink

    def __call__(self, *args, **kwargs):
        self.sink(*args, **kwargs)  # show in console
        if self.file:  # redirect to file
            if kwargs and 'file' in kwargs.keys():
                return
            with open(self.file, 'a') as f:
                self.sink(*args, file=f, **kwargs)

_print = builtins.print
log_file = os.path.join(log_dir, 'wandb', 'latest-run', 'files', 'output.log')
builtins.print = Print(file=log_file, sink=_print)

[JinchaoLove]

Hi @JinchaoLove , when running wandb offline, we save the metrics locally and associated run logs. However, the output.log, does not get save locally and requires an activate connection with our servers. There is a feature request in place for this and it is actively being review for implementation. We will update you when available.

Hi @MBakirWB , the new update in #3828 seems great! But the output.log is only generated in online mode or after we call wandb sync for the offline runs. May I know why the console logging should connect with W&B servers?

[MBakirWB]

Hi @JinchaoLove , I reviewed my comment from before and should clarify it. Wandb.init can be run in offline and online mode. Regardless of the mode, an output.log file will always be generated. Syncing offline runs will automatically sync the output logs to wandb servers. With PyTorch Lightning integration, in offline mode, the output.log isn't being generated. This appears to be a bug in the integration. I will log it with the team for review and report back when I have additional info. Thank-you.

[JinchaoLove]

Hi @MBakirWB , I see. Thanks for the kind clarification.

[exalate-issue-sync[bot]]

WandB Internal User commented: JinchaoLove commented: Hi @MBakirWB , I see. Thanks for the kind clarification.