Implement quantized sigmoid

[laurilaatu]

Quantized sigmoid function. Implementation is based on the existing quantized_tanh function. Requested in issue #59

[zhuangh]

Thanks @laurilaatu for the PR.

@lishanok @danielemoro please review. thanks!

[laurilaatu]

Thank you @laurilaatu for your contribution!

I ran the quantized_sigmoid on an input of np.linspace(-12,12,10000), but the output of the quantizer exceeded the range of [0.0, 1.0] that is expected with sigmoid. So we probably want to prevent this from happening.

I plotted the input versus output below for the quantizer versus the standard tf sigmoid function, and you can see the quantizer outputs are too large.

I believe that this occurs because I set the integer bits to a number greater than 0. But if the user ever sets the number of integers to be greater than 0, than this quantizer stops being a typical sigmoid. Also, if I do set the integer bits to be 0, you can see the quantized_sigmoid(4,0) does not look very similar to the tf sigmoid function. Perhaps we can improve this.

Thanks for the comments.

For the integer part the implementation follows that of quantized_tanh which also contains integer bits and setting integer bits in that has a similar effect on the output. For my use neither quantized_sigmoid nor quantized_tanh require the integer part. Should removing the integer part be the way to go here or is the integer part needed when doing for example integer-only quantization?

set_internal_sigmoid("real") is required for the function to follow tf.math.sigmoid. set_internal_sigmoid("real") defaults to hard_sigmoid and seems to follow that well.

[zhuangh]

Thank you @laurilaatu for your contribution!

I ran the quantized_sigmoid on an input of np.linspace(-12,12,10000), but the output of the quantizer exceeded the range of [0.0, 1.0] that is expected with sigmoid. So we probably want to prevent this from happening.

I plotted the input versus output below for the quantizer versus the standard tf sigmoid function, and you can see the quantizer outputs are too large.

I believe that this occurs because I set the integer bits to a number greater than 0. But if the user ever sets the number of integers to be greater than 0, than this quantizer stops being a typical sigmoid. Also, if I do set the integer bits to be 0, you can see the quantized_sigmoid(4,0) does not look very similar to the tf sigmoid function. Perhaps we can improve this.

To follow "return" in docstring, shouldn't the output value range of this function be [0, 1]?

[laurilaatu]

Thank you @laurilaatu for your contribution! I ran the quantized_sigmoid on an input of np.linspace(-12,12,10000), but the output of the quantizer exceeded the range of [0.0, 1.0] that is expected with sigmoid. So we probably want to prevent this from happening. I plotted the input versus output below for the quantizer versus the standard tf sigmoid function, and you can see the quantizer outputs are too large. I believe that this occurs because I set the integer bits to a number greater than 0. But if the user ever sets the number of integers to be greater than 0, than this quantizer stops being a typical sigmoid. Also, if I do set the integer bits to be 0, you can see the quantized_sigmoid(4,0) does not look very similar to the tf sigmoid function. Perhaps we can improve this.

To follow "return" in docstring, shouldn't the output value range of this function be [0, 1]?

Good point. This implementation took influence from quantized_tanh which also includes integer bits as well as output going beyond the defined values [-1,1] in docstring.

Defined here

Now it seems that defining integers is a non-wanted functionality in both sigmoid and tanh. I will remove the integers from my implementation.

[laurilaatu]

Hi @laurilaatu could you please add unittest?

Unit test added and integer part removed.

[zhuangh]

awesome! great visualization.

My question: why is the function not symmetric?

To be more specific, the absolute value of blue curve is actually higher than its quantized versions when x >3, while it is smaller when x < -3

Also, the step sizes around x=-4 and x=4 are somehow in different size. For example, the red curve.

[danielemoro]

I agree with @zhuangh. Another idea would be to set integer_bits = 1 and then clip the quantized value to [0, 1] such that we can fully cover the range of the float sigmoid function.

[laurilaatu]

awesome! great visualization.

My question: why is the function not symmetric?

To be more specific, the absolute value of blue curve is actually higher than its quantized versions when x >3, while it is smaller when x < -3

Also, the step sizes around x=-4 and x=4 are somehow in different size. For example, the red curve.

How about this symmetric implementation that uses bits to determine the quantization size 1/(2**bits) and will reach 0 and 1. After some testing this would best correspond to real sigmoid implemented as a lookup table and as the implementation uses the computation of the sigmoid function it also corresponds well to a math function based version. The red curve has unequal size of steps in x as that is the output from the sigmoid function. In smooth and hard sigmoid the step size is equal.