search

DRAGNLabs / 301r_retnet

2 stars 1 forks source link

Carbon #41

Closed DrewGalbraith closed 3 months ago

DrewGalbraith commented 3 months ago

This is a relatively small PR with the aim to add carbon-emissions tracking to the project. It simply wraps around the GPU-intensive past of the code, though it tracks use of GPUs, CPUs, TPUs, HPUs and IPUs (sorry, LPUs aren't supported yet ¯_(ツ)_/¯ ).

Anyway, there are three principle changes taking place here:

The last of these is the most intensive, but it's a pretty simple library. I'll update the README.md to reflect this feature as soon as we decide merge it (feel free to make it a Requested Revision once everything else looks good to go.)

JayOrten commented 3 months ago

Rejected, no LPU support

DrewGalbraith commented 3 months ago

I added the feature to the README.md .

DrewGalbraith commented 3 months ago

Note that commit https://github.com/DRAGNLabs/301r_retnet/pull/41/commits/a79db79ec943327f4819d8a7a73373ad42888582 ensures that all devices use NVIDIA A100s. This is necessary when running on the CS cluster as they have recently introduced H100s. This could otherwise render training times between models incomparable.