Clipping within IAF flows and out IAF flows

[zhang-jian]

Hi, In the case of use_log_scale = False, in each IAF flow, scale is clipped in the range of [tf.exp(-9.0), tf.exp(7.0)] as seen in https://github.com/bfs18/nsynth_wavenet/blob/master/wavenet/parallel_wavenet.py#L129. However, after all IAF flows, another clipping at https://github.com/bfs18/nsynth_wavenet/blob/master/wavenet/parallel_wavenet.py#L169 sets scale_tot to be minimum tf.exp(7.0). Is this not suppose to be maximum? Such as

scale_tot = tf.squeeze(tf.maximum(scale_tot, tf.exp(7.0)), axis=2)

Thanks.

[bfs18]

Hi, it is just some design choice to avoid numerical problems and keep data precision. scale_tot is the product of the scale values from all flows. If scale value from each flow is too small, scale_tot would became 0. So I choose exp(-9.0) as the minimum value for scale. Since I use 4 flows and exp(-9.0) ** 4 would not trigger data underflow in float32, scale_tot is not clipped. The answer to you question is no. I want all the scale_tot values smaller than exp(7.0), so I use tf.minimum(scale_tot, tf.exp(7.0))

[zhang-jian]

I can understand that set scale to be minimum of tf.exp(-9.0) is to avoid numerical problems. But why do we also need to set scale_tot to be minimum tf.exp(7.0), such as in line https://github.com/bfs18/nsynth_wavenet/blob/master/wavenet/parallel_wavenet.py#L169, or

scale_tot = tf.squeeze(tf.minimum(scale_tot, tf.exp(7.0)), axis=2)

Thanks

[bfs18]

The tf.minimum function compare two arrays and returns a new array containing the element-wise minima.

[zhang-jian]

Ah, confused between tf.minimum and tf.maximum. Thanks a million for the clarification.