Enable logging with intervals to save memory

nicofirst1 commented 6 months ago

Add option to allow for interval logging strategy, to save interactions every X batches.

Description

Proposed changes include:

Add IntervalLoggingStrategy (extends LoggingStrategy) that accepts an optional param log_interval. Every log_interval the strategy returns the interaction otherwise Interaction.empty().
Modified from_iterable to filter out empty interactions

Related Issue (if any)

Similar pull request

Motivation and Context

When training with limited memory resources, appending all the interactions to a list may cause OOM errors. Allowing the logging strategy to save interactions every X steps (otherwise empty) gives more flexibility.

nicofirst1 commented 6 months ago

@robertodessi What is going on with the wandb errors? I haven't imported anything new.

robertodessi commented 6 months ago

maybe a conflict in wandb versions between the CI server and EGG? I don't have access to this repo anymore, nothing much I can do about it, sorry!

nicofirst1 commented 6 months ago

Oh, I see. Thank you for the tip!

robertodessi commented 6 months ago

@nicofirst1 CI has just been deleted for this repo (look at the latest commit), it makes sense given it's not actively supported anymore

nicofirst1 commented 6 months ago

I saw... such a pity. I guess I'll use my own fork

facebookresearch / EGG