Add option for fp16 kv cache

jasonppy / VoiceCraft

Zero-Shot Speech Editing and Text-to-Speech in the Wild

Other

7.66k stars 749 forks source link

Add option for fp16 kv cache #51

Closed DeuceOfClubs closed 7 months ago

Ph0rk0z commented 7 months ago

Changed the fp32s near KV_cache to FP16 and didn't experience any loss in quality. Unfortunately the memory stayed about the same. A lot of other calculations are also done in FP32 for some reason. Haven't tried to replace all FP32 to FP16 yet.

Ph0rk0z commented 7 months ago

Where else did you change it? I added both model.half and changed the float32s to float16s. The memory can still spike.