xiph / LPCNet

Efficient neural speech synthesis
BSD 3-Clause "New" or "Revised" License
1.13k stars 295 forks source link

gatedconv.py and causalconv.py and unquantized to quantized training adaption #61

Closed J-Fo-S closed 2 years ago

J-Fo-S commented 5 years ago

Apologies in advance if I missed something obvious, but what purpose do the causalconv.py and gatedconv.py files serve? I found neither any 'causal' or 'gated' references in the paper, nor any use of these files as modules in the codebase. I assume gated is for attention and causal for future processing of the signal?

Secondly, I just realized there was a second paper on LPC and it states there are unquantized and quantized training stages/adaptation:

Training is first performed with unquantized features. When training a model for quantized features, we start with a model trained on unquantized features, and apply domain adaptation to quantized features. We have observed that better results are obtained when only the frame rate network is adapted, with the sample rate network weights left unchanged. In addition to the slightly better quality, this has the advantage of faster training for new quantizers and also smaller storage if different quantized models are needed.

@jmvalin I am uncertain if this adaptation is automated in the training code provided or if I must manually implement by training twice with adaption set to false and then true, respectively. By the parameters in the paper, it seems the latter, but I still don't see where the quantization is happening (and not) when simply setting the adaption true and false

#Set this to True to adapt an existing model (e.g. on new data)
adaptation = False

if adaptation:
    #Adapting from an existing model
    model.load_weights('lpcnet24c_384_10_G16_120.h5')
    sparsify = lpcnet.Sparsify(0, 0, 1, (0.05, 0.05, 0.2))
    lr = 0.0001
    decay = 0
else:
    #Training from scratch
    sparsify = lpcnet.Sparsify(2000, 40000, 400, (0.05, 0.05, 0.2))
    lr = 0.001
    decay = 5e-5

Any clarification would be greatly appreciated, thank you.

p.s. sorry for the empty comment - fingers slipped.