Tutorial/Smoothing algorithm is a poor choice

[bdlow]

The algorithm used in the Smoothing tutorial (https://www.arduino.cc/en/Tutorial/Smoothing) uses a simple moving average of 10 samples, storing the samples in an array.

Whilst useful in some applications, for simply removing noise from an analog input a simple moving average (IIR filter) is a more efficient and less convoluted method. e.g.:

const int filterWeight = 4;  // higher numbers = heavier filtering
const int numReadings = 10;

int average = 0;                // the average

void setup() {
  // initialize serial communication with computer:
  Serial.begin(9600);
  // seed the filter
  average = analogRead(inputPin);
}
void loop() {
  for (int i = 0; i < numReadings; i++) {
    average = average + (analogRead(inputPin) - average) >> filterWeight;
  }
  Serial.println(average);
  delay(1);
}

[q2dg]

I think introducing the shifting operator brings a extra difficulty to the example...I'm not sure if it is convenient in this case. Maybe adding this code as a "advanced method" in the same page could do the trick.

[shiftleftplusone]

IMO 2 filters make sense: 1) a low pass filter newavrg = eta * (newval) + (1-eta) * oldavrg // (e.g., eta=0.9) 2) a median filter (of 3, or of 5) by a ring memory, (although costly and lavish, but always "real values" and shoves out all extremely bad noise peaks)

[bdlow]

The shift isn't really important - that's a comparatively minor optimisation. Perhaps a note along the lines of "using a power of 2 divisor will allow the compiler to optimise the calculation".

i.e. this works just as well:

// using a power of 2 for the filterWeight will allow the compiler to optimise the calculation
const int filterWeight = 64;  // higher numbers = heavier filtering
const int numReadings = 10;

int average = 0;                // the average

void setup() {
  // initialize serial communication with computer:
  Serial.begin(9600);
  // seed the filter
  average = analogRead(inputPin);
}
void loop() {
  for (int i = 0; i < numReadings; i++) {
    average = average + (analogRead(inputPin) - average) / filterWeight;
  }
  Serial.println(average);
  delay(1);
}

[bdlow]

Ah, yes, using the phrase "low pass filter" would also be good, to at least introduce the topic and give a good term for further research.

[shiftleftplusone]

actually average = average + (analogRead(inputPin) - average) / filterWeight; is _no_ low pass filter. a low pass filter would be

average = ETA * (analogRead(inputPin)) + (1-ETA) * average; // ETA: filter weight, e.g. ETA = 0.9

[PaulStoffregen]

Those 2 equations are identical, if 1/filterWeight is ETA. Simple algebra to factor and rearrange the terms.

[bdlow]

Indeed, same function. The 'filterWeight' version lends itself directly to a pure-integer shift-rather-than-division implementation; the traditional expression typically has 'ETA' (alpha) as a fraction (real).

[PaulStoffregen]

Agreed, from a practical numerical viewpoint, the original is far better.

But for signed division, the complier usually doesn't replace powers of 2 by shifting. That tends to happen only for unsigned numbers.

[shiftleftplusone]

no, it's not identical IMO, because average is multiplied by (1-ETA) and aditionally (!) newvalue is multiplied by ETA. I don't see a multiplicant by your newvalue. But I stand corrected in case I'm wrong.

anyway, average = ETA * (newvalue) + (1-ETA) * average; is the common formula for lowpass filters.

[PaulStoffregen]

sigh Basic algebra....

average = ETA * (newvalue) + (1-ETA) * average ETA = 1 / filterWeight average = (1 / filterWeight) * newvalue + (1 - 1 / filterWeight) * average average = newvalue / filterWeight + average * (1 - 1 / filterWeight) average = newvalue / filterWeight + average - average / filterWeight average = average + newvalue / filterWeight - average / filterWeight average = average + (newvalue - average) / filterWeight

[shiftleftplusone]

you win! :) but nevertheless, my formula is better : average = ETA * (newvalue) + (1-ETA) * average because it's easier, and no division by zero possible, and it's the common formula notation,and because there will be no integer division fraction/cut-off/rounding error or how ever you may call this in English 8-)

[NicoHood]

I also like his formular, although I dont know what ETA means, but i know what it is used for.

I also use this here: https://github.com/NicoHood/MSGEQ7/blob/dev/src/MSGEQ7.hpp#L135 I just wasnt sure if there was a better implementation for it.

[shiftleftplusone]

float ETA is just a weight factor (don't know how to say it better in English). 0<= ETA <=1

by ETA=0.5 its just the arithmetic average (mean), by 0.9 the current readings are more strongly weighted, and by ETA < 0.5 it's just an extremely idle/dull filter, showing almost no peaks any more.

[NicoHood]

Sure. I just wonder what E, T and A stand for. What is does i know for sure ;)

[shiftleftplusone]

eta is a Greek letter (alpha, beta, gamma, delta, epsilon, zeta, eta, theta,....) and is often used im mathematics for constants in analysis.

As it's a constant, I'm writing it in Capital letters as usual for constants in C programs.

[NicoHood]

I never got above epsilon :D But yeah this function should do it. But please use uint16_t values here, no floating points. (see link above, i do exactly the same)

[shiftleftplusone]

In New Common Greek it's pronounced "iita", not "eeeta" as in Ancient Classic Greek.

But you may use float EPSILON; instead as well - but admittedly you'd have much more chars to write... ;)

Nevertheless, as ETA is from 0...1 only floats will work. In the _real_ world nothing is integer, every number is _real_, i.e. (rational, irrational, or even transcendental) a _float_. ;)

[Arjan-Woltjer]

@Vogonjeltzz, I don't know what is better. As Paul already mentioned, his solution can be implemented by bitshift, which is far more economical than your solution. So on paper you may win, in practice Paul is using processing power far better.

[shiftleftplusone]

ok, for speed on an AVR you are right, but for a tutorial I would prefer the float version I posted nevertheless. I'm personally using Dues above all, and so the speed by float calculations is no issue generally .

[PaulStoffregen]

FWIW, I don't deserve any credit for the original code, even though I've independently used that same equation in several projects over the years.

When evaluating these algorithms, with either integers or floats, the effects of limited precision should be investigated. Especially important is the case of a small AC or higher frequency signal with a large DC or very low frequency offset. The practical effect of limited numerical precision in the addition or subtraction is a point where special attention should be paid.

I would caution anyone reading this to avoid investing too much confidence in arguments that amount to personal preference, without actual testing or rigorous mathematical analysis.

[shiftleftplusone]

for educational reasons, the approach by using floats is more intuitive, and additionally, always precise and accurate (related to the chosen value of ETA and the precision of 32bit floats in general):

float average, ETA=0.9;
average = ETA * (float)newvalue + (1-ETA) * average

For filters (of either kind), in general, by a scientific+ statistical point of view, the sensitivity, accuracy, and the evidence has to be proven of course by a statistical evaluation. E.g. a low-pass filter may severly suffer from bad (e.g., "rectified") noise by independent high non-Gaussian peaks which will badly affect the long-time average. In this case a median filter would be more suitable. But this aspect will overstrain the intentions of a smoothing sensor tutorial for beginners, IMO.

[PaulStoffregen]

using floats is more intuitive, and additionally, always precise and accurate

That's quite an overly broad and confident assumption! Perhaps the word "always" is meant to include ARM chips, where "int" is 32 bits, and "float" uses a 23 bit mantissa?

Again, I would caution anyone reading this thread from placing too much faith in some of these arguments, which amount to little more than casual conversation about personal preferences.

[shiftleftplusone]

no, "always precise and accurate" is related to the range of expectable results and digits over the whole range of floats even for integer parts ("Vorkommastellen") <1.0 or even <10.0, still providing 6-7 fraction digits, and it's related to the fact that no division by zero is possible which would result in nans ( related to average = average + (analogRead(inputPin) - average) / filterWeight;)

Even more precise would be using double precision floats though. And remember that an Arduino tutorial is not just an AVR tutorial but also has to target 32-bit ARM or Atheros cpu users as well, which is becoming more an more significant and frequent.

Anyway, (analogRead(inputPin) - average) >> filterWeight is a term which is absolutely misplaced in a beginner's tutorial.

[NicoHood]

I never ever used a single float in any of my c/c++ programs and I probably never will unless I code for a PC and then I will avoid it anyways. If you ask me AVR should not support float at all because its just not the right platform to use it. But a simple smooth value from 0-255 is totally fine i'd say, which can be achieved with the formular too.

[shiftleftplusone]

yes, sure it's possible, and there are many more filters possible, too, e.g. just a mean average or a median filter or even a Kalman filter or a MonteCarlo filter.

But this issue/ topic is about a tutorial of a low pass filter (CMIIW), and therefore the point is about an educational approach to make it capable for beginners.

Floats or not floats is a complete different issue - to me I'm always using floats because my programs require floats. Just everyone to his special needs.

[NicoHood]

Just everyone to his special needs.

Yep have fun XD

I'll leave the choice to @agdl or maybe @tigoe can help here.

[tigoe]

The original tutorial isn't the most perfect averaging algorithm, but it has worked for a truckload of beginners. I'm inclined to leave it in place.

That said, I wouldn't mind if people wanted to contribute a few filter examples, aimed at beginners. I could see the value of having a set of them. @karlward was working on a filter library a while back: https://github.com/karlward/filter but moved on to other things.

I quite like the one @VogonJeltz and @NicoHood have going there:

float confidence = 0.5;
float average;

void setup() {
 Serial.begin(9600);
}

void loop() {
  int sensorReading = analogRead(A0);
  average = confidence * (sensorReading) + (1 - sensorReading) * average;
  Serial.print(sensorReading);
  Serial.print(",");
  Serial.println(average);
}

I ran it with an accelerometer and graphed a few seconds' worth of readings three or four times and was very pleased with the result. Not very scientific, but for a quick test, better than nothing. It only introduces two concepts above analogRead(): floats and the confidence factor (filter weight). So it'd be relatively easy to explain to beginners. What I like about it over @bdlow's original implementation is that in this version, you don't have to explain why you're taking ten samples. Every sample is weighted into the running average.

I'd be happy to change to this if @agdl and @damellis have no objection.

[shiftleftplusone]

sorry, not average = confidence * (sensorReading) + (1 - sensorReading) * average;

moreover, the expression "confidence" is misleading and ambiguous here IMO.

but average = filterweight * (sensorReading) + (1 - filterweight ) * average; // filterweight == ETA, 0=<ETA<=1

would be fine.

[tigoe]

I was looking for an everyday word to explain what filterWeight means. If you were explaining it to someone unfamiliar with the math, what word would you use?

[shiftleftplusone]

filterweight

[shiftleftplusone]

or smoothingweight ;)

[shiftleftplusone]

anyway, even a junior high is supposed to know what a constant of either name does when multiplied with a value. But IMO it shuldn't ever be suggestive by "everxdameaning" like "confidence" whi is also sth iike "sanguinely"

[tigoe]

Yes, but you'd be surprised to know how many Arduino users stopped caring about math because some junior high school teacher told them they're "supposed to know". Have a little empathy here for someone with different intellectual talents. Hasn't someone else ever made you feel stupid?

This is the kind of thing beginners trip over. Give me a word you'd use to describe what it means, not the word itself. I call it confidence because it reflects how confident I am that the sensor picked up a legitimate value and not some random electrical noise. How would you explain it so someone you care about, but who is terrible at math, for example?

[shiftleftplusone]

no- not about "filterweight" but sanguinely, confident, trustful, friendly for a variable name is highly misleading and ambiguous.

[shiftleftplusone]

you should explain: filterweight is a constant factor for smoothing. use it around 0.9 for a little smoothing, 0.7 for more smoothing and even lower for very heavy smoothing

(english is not my native language, you're supposed to know what I mean nevertheless)

[tigoe]

Okay. You want me to play devil's advocate until we get the terms, I can do that.

What is a constant factor?

Why 0.9 and not, say, 32?

Why 0.7 instead of, say, -1?

What do you mean by weight?

[shiftleftplusone]

it's 0..1 by definition. Otherwise it won't work

[tigoe]

Who defined it and why did they define it that way?

[shiftleftplusone]

the author of the filter defined it for the filter to work. But that's a really stupid attitude. In maths it is as it is, period.

people who don't understand even that better don't start programming, at least no filters.

using the term "confidence" instead of "filterweight" wouldn't make it better - just the opposite. Maths must not be suggestive, everything has to be just as it is by definition.

[q2dg]

It's a number which gives some elements more "weight" or influence on the result than other elements in the same set (from Wikipedia). Its value goes from 0 -cancelling the presence of that specific element- to 1 -being transparent-. For instance: a value of 0.5 will give to associated element halt the importance of the others.

2015-10-12 19:36 GMT+02:00 tigoe notifications@github.com:

Who defined it and why did they define it that way?

— Reply to this email directly or view it on GitHub https://github.com/arduino/Arduino/issues/3934#issuecomment-147471026.

[tigoe]

I disagree. I see 110new students every year, graduate students, and a good 20% of them are interested in programming, but have been turned off by math sometime in the past. Often it's because people told them they "should know". We made Arduino mainly to make it possible to explain programming to these kinds of people. And it is possible. I've seen it work.

But it requires that we let go of the notion that someone "should know". I can show you engineering students who can run rings around many people in this thread but wouldn't know a gerund from a whole in the ground. And when they're in a room with writers who can't program, and they both let go of their assumptions and trust each other, they teach other and they both learn. That is why I'm pushing on this. I want to help people feel like they understand technology, not just those who are good at it.

@q2dg, thank you. That is a much more usable and succinct answer.

[shiftleftplusone]

well, do want you think what would be best. I was teaching lyceum/highschool students of 10-19 years age and I'm quite sure what they are capable of or not (at least in germany) ;) I honestly don't see what in a formula like

average = filterweight * (sensorReading) + (1 - filterweight ) * average; // choose filterweight 0...1

could be misleading.

But I know for sure what would be misleading using "confidence", especially for non-English speakers. you might also call it "prayer", with the same result.

[NicoHood]

average = filterweight * (sensorReading) + (255 - filterweight ) * average; 0-255 and the program is fast. How come that? \o/ IF you decide to work on a microcontroller, then please do it properly from the beginning.

Basic math guys, basic math... I think students in that age will understand that. If not you are a bad teacher if you are not able to explain that in a minute. And if they dont understand they are in the wrong class or should read the if - else section again.

[shiftleftplusone]

do what you want, I only can give advices. I don't care about AVRs and their limitations, my Due is closely to fast enough, and a Teensy or a Zero or a Yun also is supposed to be. I don't care about embedded systems by rediculous performance not being able to handle floats., and handling of floats is indispensible and a minimum requirement (such as sin, cos, atan, sqrt, pow).

[tigoe]

Sorry, @VogonJeltz, I didn't mean to insult your experience. I get agitated on this issue.

@q2dg laid it to rest for me above. I'm fine using "filterWeight" as long as the comments at the top of the sketch read something like that Wikipedia quote. Modified, I'd say:

"filterWeight is a measure of the importance or "weight" of one sensor reading relative to the others. A reading's weight is measured on a scale from 0 (i.e. no value at all) to 1 (i.e. more important than all the others)." (I think I got that backwards, but that's the general idea.)

I think what you said about 0.7 - 0.9 is useful, but I want to find more explicit language for it.

[shiftleftplusone]

first comes the equation, then comes the explanation. You might wish to work out the explanation, from time to time, as time goes by....

[shiftleftplusone]

The filter recurrence relation provides a way to determine the output samples in terms of the input samples and the preceding output. The following pseudocode algorithm simulates the effect of a low-pass filter on a series of digital samples:

// Return RC low-pass filter output samples, given input samples, // time interval dt, and time constant RC function lowpass(real[0..n] x, real dt, real RC) var real[0..n] y var real α := dt / (RC + dt) y[0] := x[0] for i from 1 to n y[i] := α * x[i] + (1-α) * y[i-1] return y

The loop that calculates each of the n outputs can be refactored into the equivalent:

for i from 1 to n y[i] := y[i-1] + α * (x[i] - y[i-1])

That is, the change from one filter output to the next is proportional to the difference between the previous output and the next input. This exponential smoothing property matches the exponential decay seen in the continuous-time system. As expected, as the time constant \scriptstyle RC increases, the discrete-time smoothing parameter \scriptstyle \alpha decreases, and the output samples \scriptstyle (y_1,\, y_2,\, \ldots,\, y_n) respond more slowly to a change in the input samples \scriptstyle (x_1,\, x_2,\, \ldots,\, x_n); the system has more inertia. This filter is an infinite-impulse-response (IIR) single-pole low-pass filter.

https://en.wikipedia.org/wiki/Low-pass_filter#Discrete-time_realization

[tigoe]

Right. I'll come back and turn that into something more digestible later on.

[shiftleftplusone]

remember, it's a formula. call the terms and multiplicants as you want but keep away from being suggestive / evocative. a factor is a factor is a factor. Explain what it does by which effecs, that's the simple point.

[tigoe]

Clearly we have different ways of teaching. I've found that being suggestive works very well, when the suggestion is in the students' experience. I spend much of my time looking for the right analogies.