search

CUNY-CL / yoyodyne

Small-vocabulary sequence-to-sequence generation with optional feature conditioning
Apache License 2.0
25 stars 15 forks source link

Implement Pytorch Metrics #158

Open bonham79 opened 4 months ago

bonham79 commented 4 months ago

TorchMetrics support is pretty reliable nowadays and makes distributed training less annoying (no more World sizes, yay!). It also syncs well with Wandb logging and allows monitoring of training batch performance. Any complaints about me migrating validation logging to this?

kylebgorman commented 4 months ago

I don't know anything about it but it sounds like a positive.

bonham79 commented 4 months ago

It's just a suite of metrics that torch sets up to manage multi-gpus under the hood. You just pass it your tensors during the training loop and it will store submetrics. Then when you need the actual metric you call and it does the calculation and memory collection under the hood. It saves you from desync issues if you have more than GPU for training. Also PTL supports it so it helps reduce boilerplate for other metrics.

Adamits commented 4 months ago

I think I tried to do this last year, and had some issues getting the features I actually wanted from it to track, so I gave up, but I think it was in beta or something then. I'd be happy if you got it working :)

bonham79 commented 4 months ago

I just got it running for BLEU scores on another project, so CER/WER should be stable by now.