search

sfilip / fquantizer

Euclidean lattice-based quantization routines for digital filter design
GNU General Public License v3.0
1 stars 1 forks source link

failure to converge when several bands are specified. #5

Open jlwehle opened 5 years ago

jlwehle commented 5 years ago

Attach is a test case based on a CIC compensation filter where fquant succeeds if the number of frequency bands is reduced below a certain threshold, otherwise it fails. The code is built like so:

clang++ -I/usr/local/include -O2 -march=amdfam10 \
  -o fquant_tc fquant_tc.cxx -L/usr/local/lib -lfquantizer -lmpfr

and running it results in:

Parks-McClellan delta = 0.017878909509322568616
  Naive rounding error    = 0.84135723114013671875

  Reduction = 600.09815000000003238 ms
  LLL initial error       = 0.85398578643798828125
  Band 0 error = 0.85398578643798828125
  Band 1 error = 0.24652576703809714331
  Band 2 error = 0.029929713393439527636
  Band 3 error = 0.13246696585733630513
  Band 4 error = 0.41661805504400462041
  Band 5 error = 0.21240771528807655955
  Band 6 error = 0.018072934649282501596
  ...
  The exchange algorithm did not converge.
  TRIGGER: Not enough alternating extrema!
  POSSIBLE CAUSE: Nmax too small
  The exchange algorithm did not converge.
  TRIGGER: numerical instability
  POSSIBLE CAUSES: poor starting reference and/or a too small value for Nmax.
  Parks-McClellan delta = nan
[fquant_tc.cxx.txt](https://github.com/sfilip/fquantizer/files/3823011/fquant_tc.cxx.txt)

It would be useful for more cases to work with fquant.

-- John