Closed kongzijian closed 8 months ago
trainging set is : dataloader = DataLoader(dataset1, num_workers=0, batch_size=batch_size, shuffle=True) from pytorch_lightning.callbacks.progress import TQDMProgressBar trainer = pl.Trainer(gpus=n_gpus, strategy="ddp", precision=16, accelerator="gpu", callbacks=[TQDMProgressBar(refresh_rate=1)], accumulate_grad_batches=accumulate_grad_batches,log_every_n_steps=20)
There may be some bugs for your dataloader. We add "try except" in our dataset base.py to skip some samples. You could remove the try except to debug.
trainging set is : dataloader = DataLoader(dataset1, num_workers=0, batch_size=batch_size, shuffle=True) from pytorch_lightning.callbacks.progress import TQDMProgressBar trainer = pl.Trainer(gpus=n_gpus, strategy="ddp", precision=16, accelerator="gpu", callbacks=[TQDMProgressBar(refresh_rate=1)], accumulate_grad_batches=accumulate_grad_batches,log_every_n_steps=20)