Quantization may output more levels than the number of bits allows

[baggepinnen]

The following model demonstrates that the quantization function may output more levels than the number of bits allows, which is $2^\text{bits}$. The quantization is here set to 2 bits, allowing for 4 distinct output levels However, the number of unique output levels is 5:

Full model, implemented with OpenModelica

OpenModelica 1.24.0~dev-16-gaa27cc1
OMSimulator 2.1.2-linux-notlm-debug

model quanrt
  Modelica.Clocked.RealSignals.Sampler.SampleWithADeffects sample1(limited = true, quantized = true, bits = 2)  annotation(
    Placement(transformation(origin = {0, 22}, extent = {{-6, -6}, {6, 6}})));
  Modelica.Blocks.Sources.Sine sine1(f = 1)  annotation(
    Placement(transformation(origin = {-28, 22}, extent = {{-10, -10}, {10, 10}})));
  Modelica.Clocked.ClockSignals.Clocks.PeriodicRealClock periodicClock1(period = 0.01)  annotation(
    Placement(transformation(origin = {18, 2}, extent = {{-6, -6}, {6, 6}})));
  Modelica.Clocked.RealSignals.Sampler.AssignClock assignClock1 annotation(
    Placement(transformation(origin = {36, 22}, extent = {{-6, -6}, {6, 6}})));
equation
  connect(sine1.y, sample1.u) annotation(
    Line(points = {{-17, 22}, {-7, 22}}, color = {0, 0, 127}));
  connect(periodicClock1.y, assignClock1.clock) annotation(
    Line(points = {{25, 2}, {36, 2}, {36, 14}}, color = {175, 175, 175}));
  connect(sample1.y, assignClock1.u) annotation(
    Line(points = {{7, 22}, {28, 22}}, color = {0, 0, 127}));
  annotation(
    uses(Modelica(version = "4.0.0")));
end quanrt;

[HansOlsson]

The underlying models seems to have several issues:

• An off-by-one error as shown by this model. Resolution is the difference between levels, so if we want 2^bits levels there are 2^bits-1 differences.
• Complicated and broken logic:
  • Having abs and sign-information seems weird, especially when allowing non-symmetric limits, e.g., set yMin=-1.1 and yMax=1.0; and see that the quantized result goes from -1.1 to +1.1 (which exceeds the limit)!
  • In general having the limits before quantization has the risk that quantization breaks the limits.

A better variant seems to be: y = resolution*floor((u-yMin)/resolution+0.5)+yMin; and parameter Real resolution = if quantized then ((yMax - yMin)/(2^bits-1)) else 0;, but I don't know how such devices are normally built.

Note that the problem gets worse as the number of bits decreases; and it seems no-one counted the 257 levels for the default setting.

[HansOlsson]

As expected the DSP world is a bit more complicated.

What I proposed is closer to a mid-riser one https://en.wikipedia.org/wiki/Quantization_(signal_processing) - whereas the current formula seems to be broken variant of a dead-zone quantizer (a form of mid-tread quantizer).

However, having non-symmetric limits isn't considered on that page; and I assume most people who use a non-symmetric limit in practice use it for a non-negative limit (such as yMin=0, yMax=255).