Closed morestart closed 2 years ago
morganmcg1
commented
2 years ago @morestart can you please share some code to reproduce this? And describe what you were trying to log (e.g. what task/data type etc)
Also make sure you are on the latest version of wandb: pip install wandb --upgrade
morestart
commented
2 years ago this is my code @morganmcg1,my wandb version is 0.12.6,This error occurs in epoch 69 now i run a new program,It works well... the error info is too little, i don't know why..
class DetModel(pl.LightningModule):
def forward(self, x):
features = self.encoder(x)
features = self.neck(features)
features = self.head(features)
return features
def training_step(self, batch, batch_idx):
data = batch
output = self.forward(data['img'])
loss_dict = self.loss_func(output, batch)
self.log(name=self.train_loss_name, value=loss_dict['loss'])
self.log(name='shrink_maps', value=loss_dict['loss_shrink_maps'])
self.log(name='threshold_maps', value=loss_dict['loss_threshold_maps'])
self.log(name='binary_maps', value=loss_dict['loss_binary_maps'])
return loss_dict['loss']
def validation_step(self, batch, batch_idx):
data = batch
output = self.forward(data['img'])
boxes, scores = self.postprocess(output.cpu().numpy(), batch['shape'])
raw_metric = self.metric(batch, (boxes, scores))
return raw_metric
def validation_epoch_end(self, outputs):
metric = self.metric.gather_measure(outputs)
self.log('recall', value=metric['recall'].avg)
self.log('precision', value=metric['precision'].avg)
self.log('hmean', value=metric['fmeasure'].avg)
return {'hmean': metric['fmeasure'].avg}
def configure_optimizers(self):
optimizer = get_optimizer(self.parameters(), self.optimizer_name, self.lr)
lr_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max')
return {'optimizer': optimizer, 'lr_scheduler': lr_scheduler, "monitor": 'hmean'}
stale[bot]
commented
2 years ago This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!
borisdayma
commented
2 years ago This should now be fixed. Could you update Pytorch Lightning from master branch and install the most recent version of wandb?
pip install --upgrade wandb
pip install --upgrade git+https://github.com/PytorchLightning/pytorch-lightning.gitok i will try it and close this issue~thanks for your work!
🐛 Bug
Exception: The wandb backend process has shutdown
full error info:
To Reproduce
Expected behavior
Environment
conda,pip, source): condatorch.__config__.show():cc @awaelchli @morganmcg1 @AyushExel @borisdayma @scottire