Progress of generated sounds

[lukysummer]

Thank you so much for an amazing paper and repository.

I have built waveGAN model in PyTorch (https://github.com/lukysummer/WaveGAN-in-PyTorch), and currently training it for the articulation of number "7". So I am just using all "Seven_*.wav" files in sc09/ as training inputs. I am currently at around Epoch 30, having sampled a generated audio every 5 epochs. But the pattern I am finding is that all six generated audios so far show very fast spiky repetition of drumy sounds such as this waveform:

Did this ever happen early in your training stage? If not, did you see any particular pattern building up in the generated audio (output of Generator) as training progressed?

Also, would the Discriminator and Generator's weights have to be initialized at all?

Thank you so much.

[chrisdonahue]

Thanks for your interest!

That output is bizarre. The only time I've ever seen something like that is when I tried training a WaveGAN using a frequency-domain discriminator. The time-domain generator learned to create little "grains" of sound which had the right spectral properties to fool the discriminator but were not very listenable.

One possibility is that you might be discriminating on "patches" of the output rather than the full output. Are you aggregating the final convolutional output of the discriminator to a single scalar output? If you are operating on patches (certain approaches like pix2pix intentionally do this), the generator might only be trying to make coherent patches but have no context of the entire waveform.

The training pattern I observed mostly started with noisy sounds and became more and more speech like with less noise as training progressed.

Discriminator and generator's weights should be initialized to random noise. I used Tensorflow's default weight initialization, which I believe defaults to "Glorot uniform initialization"

[lukysummer]

Thank you for your comment.

I think the bizarre output seems to be due to the shape of sc09 training set. Since most of the sc09 .wav files are shorter than the desired slice length of 16384, which type of padding did you use for the real data input x? I used "edge" padding only on the right side, so just repeating the last sample's value for 16384 - len(signal) times at the end.

Also, for loader.py in your wavegan repository, in the following code segment:

(Randomize starting phase:) if slice_randomize_offset: start = tf.random_uniform([], maxval=slice_len, dtype=tf.int32) audio = audio[start:]

Were you proposing a phase shuffle of random value between [0, 16384] for the real data x?

I noticed that when I trained with drums/ training set, where all the .wav files are exactly 16384 samples, this bizarre output pattern did not occur...

Thank you again!

[lukysummer]

Actually, the model started working extremely well (recognizable articulation of digits 0-9 as early as Epoch 30) when I changed the model dim (d) to 32 from 64!

[chrisdonahue]

To pad the SC09 digits to 16384 samples, I used zero padding at the end of the audio files. This is more typical for audio signal processing than the "edge" padding you described (though for this dataset they are approximately the same)

Your interpretation of the code snippet you mentioned is correct. It chooses a random offset into the waveform to produce training examples. However, I do not use this option for the digits dataset nor the drum sound effects, only for the more variable-length datasets like music or bird sounds.

Surprised to hear things improved when you reduced the size of the model. But I'm glad to hear that something worked :)