search

bytedance / lightseq

LightSeq: A High Performance Library for Sequence Processing and Generation
Other
3.22k stars 329 forks source link

Add gradient communication quantization (GCQ) #367

Closed HandH1998 closed 2 years ago

HandH1998 commented 2 years ago

I make a new directory at lightseq/training/gradient_comm_quantization, where I put GCQ code. The main change is at trainer.py, where I register communication hook to conduct GCQ.

HandH1998 commented 2 years ago

I make a new directory at lightseq/training/gcq, where I put GCQ code. GCQ has supported huggingface and fairseq.

HandH1998 commented 2 years ago

I have optimized GCQ code for Fairseq and Hugging Face.

HandH1998 commented 2 years ago

Optimize code again.

HandH1998 commented 2 years ago

Add torch version check.