Not exactly a bug but, when running offline wandb stores the new .csv files of inference to your .local and .cash folders, and that's quite a lot of data (I got 60G before my storage maxed out and broke training). Which is not exactly criminal but is quite a lot of space so we probably want to cut it down or at least be able to say "stop" somehow.
Suggested fix
Assuming we can forgo having inference at every 50 steps, I'm looking into only keeping best epoch results and cleaning up all else; but all suggestions welcome
Or, I guess, we can just always run wandb online for immediate sync without storing locally, but it feels wrong
Describe the bug
Not exactly a bug but, when running offline wandb stores the new .csv files of inference to your .local and .cash folders, and that's quite a lot of data (I got 60G before my storage maxed out and broke training). Which is not exactly criminal but is quite a lot of space so we probably want to cut it down or at least be able to say "stop" somehow.
Suggested fix
Assuming we can forgo having inference at every 50 steps, I'm looking into only keeping best epoch results and cleaning up all else; but all suggestions welcome Or, I guess, we can just always run wandb online for immediate sync without storing locally, but it feels wrong