Why predictions's time step is different in MB-MelGAN?

[ymzlygw]

As follows: ex1: ex2:

generated speech is always 2000 longer than ground-truth speech. Is this the correct prediction?

[dathudeptrai]

@ymzlygw can you share your mb-melgan config ?

[ymzlygw]

@ymzlygw can you share your mb-melgan config ? @dathudeptrai Ok, Here is:

sampling_rate: 16000 hop_size: 200 # Hop size. 256 for 22050HZ. 200 for 16KHZ format: "npy"

###########################################################

GENERATOR NETWORK ARCHITECTURE SETTING

########################################################### model_type: "multiband_melgan_generator"

multiband_melgan_generator_params: out_channels: 4 # Number of output channels (number of subbands). kernel_size: 7 # Kernel size of initial and final conv layers. filters: 384 # Initial number of channels for conv layers. upsample_scales: [8, 4, 2] # List of Upsampling scales. stack_kernel_size: 3 # Kernel size of dilated conv layers in residual stack. stacks: 4 # Number of stacks in a single residual stack module. is_weight_norm: false # Use weight-norm or not.

###########################################################

DISCRIMINATOR NETWORK ARCHITECTURE SETTING

########################################################### multiband_melgan_discriminator_params: out_channels: 1 # Number of output channels. scales: 3 # Number of multi-scales. downsample_pooling: "AveragePooling1D" # Pooling type for the input downsampling. downsample_pooling_params: # Parameters of the above pooling function. pool_size: 4 strides: 2 kernel_sizes: [5, 3] # List of kernel size. filters: 16 # Number of channels of the initial conv layer. max_downsample_filters: 512 # Maximum number of channels of downsampling layers. downsample_scales: [4, 4, 4] # List of downsampling scales. nonlinear_activation: "LeakyReLU" # Nonlinear activation function. nonlinear_activation_params: # Parameters of nonlinear activation function. alpha: 0.2 is_weight_norm: false # Use weight-norm or not.

###########################################################

STFT LOSS SETTING

########################################################### stft_loss_params: fft_lengths: [1024, 2048, 512] # List of FFT size for STFT-based loss. frame_steps: [120, 240, 50] # List of hop size for STFT-based loss frame_lengths: [600, 1200, 240] # List of window length for STFT-based loss.

subband_stft_loss_params: fft_lengths: [384, 683, 171] # List of FFT size for STFT-based loss. frame_steps: [30, 60, 10] # List of hop size for STFT-based loss frame_lengths: [150, 300, 60] # List of window length for STFT-based loss.

###########################################################

ADVERSARIAL LOSS SETTING

########################################################### lambda_feat_match: 10.0 # Loss balancing coefficient for feature matching loss lambda_adv: 2.5 # Loss balancing coefficient for adversarial loss.

###########################################################

DATA LOADER SETTING

########################################################### batch_size: 64 # Batch size for each GPU with assuming that gradient_accumulation_steps == 1. valid_batch_size: 2 batch_max_steps: 8192 # Length of each audio in batch for training. Make sure dividable by hop_size. batch_max_steps_valid: 8192 # Length of each audio for validation. Make sure dividable by hope_size. remove_short_samples: true # Whether to remove samples the length of which are less than batch_max_steps. allow_cache: False # Whether to allow cache in dataset. If true, it requires cpu memory. is_shuffle: False # shuffle dataset after each epoch.

###########################################################

OPTIMIZER & SCHEDULER SETTING

########################################################### generator_optimizer_params: lr_fn: "PiecewiseConstantDecay" lr_params: boundaries: [100000, 200000, 300000, 400000, 500000, 600000, 700000] values: [0.0005, 0.0005, 0.00025, 0.000125, 0.0000625, 0.00003125, 0.000015625, 0.000001] amsgrad: false

discriminator_optimizer_params: lr_fn: "PiecewiseConstantDecay" lr_params: boundaries: [100000, 200000, 300000, 400000, 500000] values: [0.00025, 0.000125, 0.0000625, 0.00003125, 0.000015625, 0.000001] amsgrad: false

gradient_accumulation_steps: 1 ###########################################################

INTERVAL SETTING

########################################################### discriminator_train_start_steps: 200000 # steps begin training discriminator train_max_steps: 4000000 # Number of training steps. save_interval_steps: 50000 # Interval steps to save checkpoint. eval_interval_steps: 20000 # Interval steps to evaluate the network. log_interval_steps: 200 # Interval steps to record the training log.

###########################################################

OTHER SETTING

########################################################### num_save_intermediate_results: 1 # Number of batch to be saved as intermediate results.

[dathudeptrai]

@ymzlygw hop_size should be 256

[ymzlygw]

@ymzlygw hop_size should be 256 @dathudeptrai While, My sample_rate is 16kHz, hop_size=sample_rate * 0.0125, not this?

[dathudeptrai]

@ymzlygw no, 0.016 :D

[ymzlygw]

@ymzlygw no, 0.016 :D

@dathudeptrai But after I set hop_size=256, and train with 200k step, the prediction fig is strange compared with hop_size=200 at 200k step: step=200k, hop_size=200 step=200k, hop_size=256

It seems that generator is not well trained after 200k steps. So for now , I got this: When sr=16kHz, hop_size=200, the generator can be well trained but generated audio speed is slower than truth audio and prediction have 2000 more steps. When sr=16kHz, hop_size=256, the generator can not be well trained. Is there other config that should be changed? For example, up_sample scalse?

[ymzlygw]

@dathudeptrai Maybe set hop_size=200 is correct for training with sr=16000hz. But it should change some others's config to correct the time step (there are 2000 time-steps wrong in prediction that cause generated audio is speed slow than raw auido)

[engineergun]

I had the same problem and @dathudeptrai said https://github.com/TensorSpeech/TensorFlowTTS/issues/676#issuecomment-927125526 then it was solved. Additionally, according to multiband melgan paper for 16khz data hopsize is 200 and upsample scales are 2,5,5.

[ymzlygw]

I had the same problem and @dathudeptrai said https://github.com/TensorSpeech/TensorFlowTTS/issues/676#issuecomment-927125526 then it was solved. Additionally, according to multiband melgan paper for 16khz data hopsize is 200 and upsample scales are 2,5,5.

@engineergun Thanks! Let me check it!

[ymzlygw]

@engineergun Now I am change the config: sample_rate=16kHz, hop_size=200, up_scasle=[2,5,5]. And begin training it, if there are other problems or preprocess I'll tell you. Thanks for your reply!

[engineergun]

@engineergun Now I am change the config: sample_rate=16kHz, hop_size=200, up_scasle=[2,5,5]. And begin training it, if there are other problems or preprocess I'll tell you. Thanks for your reply!

@ymzlygw you'r welcome. i hope get good results :)

[OnceJune]

mbmelgan used 4 bands in pqmf then you should make hop_size / num_bands = multiple all up_scales e.g. 200 / 4 = 2 5 5, so you set up_scales=[2, 5, 5]

[ymzlygw]

mbmelgan used 4 bands in pqmf then you should make hop_size / num_bands = multiple all up_scales e.g. 200 / 4 = 2 5 5, so you set up_scales=[2, 5, 5]

Thanks for your reply! I have changed it and now it looks work fine. I will close the issue.