Update UNET3D - Githubissues

mlcommons / training

Reference implementations of MLPerf™ training benchmarks

https://mlcommons.org/en/groups/training

Apache License 2.0

1.58k stars 549 forks source link

Update UNET3D #458

Closed mmarcinkiewicz closed 3 years ago

mmarcinkiewicz commented 3 years ago

Fixed global batch size and samples per epoch dependency on gradient accumulation
Added support for LastBatchPadded
Added logging for weight initialization
Corrected logging (global batch size, training samples, gradient accumulation steps)
Updated readme with agreed evaluation schedule

github-actions[bot] commented 3 years ago

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

johntran-nv commented 3 years ago

@mndevec , @mmarcinkiewicz , are we good to merge this one?

mmarcinkiewicz commented 3 years ago

As per conversation in the email - I reverted back the samples and left only the fix for gradient accumulation and additional logging. Today we have a meeting and I expect to have a final green light (or last minute corrections) to have it merged.

mmarcinkiewicz commented 3 years ago

I updated the reference code with the changes we discussed. @mndevec please review them as soon as possible. Pay special attention to update in readme, LR warmup/decay schedules, and how the progress is logged. They are aligned with the results of our discussion.

mmarcinkiewicz commented 3 years ago

@johntran-nv we have an approval