Issue with dew point calculation

[MarkCiliaVincenti]

If I provide a value of say temperature of 17.9C with 100% humidity, the dew point is returning a value of 17.8C when the expected would be 17.9C. It wasn't always like this and I don't have time to investigate better at the moment.

[pgrawehr]

Well, according to https://en.wikipedia.org/wiki/Dew_point the calcuation is an approximation anyway, and I would doubt it is actually defined for 100% RH.

[MarkCiliaVincenti]

Quoting from the page you linked:

A relative humidity of 100% indicates the dew point is equal to the current temperature and that the air is maximally saturated with water.

[pgrawehr]

I'm absolutely not doubting that this should be the case in theory. I'm just saying that the applied formula is an approximation that has a certain error. The error changes linearily over the temperature range, for 100% RH, we get the following results (temperatures in Degrees Celsius, rounded to 2 digits):

Input	Output
-20	-20.08
-10	-10.02
0	0.00
10	9.93
20	19.90
30	29.77
40	39.58

I guess changing the constant set used (there are different possible constant sets proposed) could improve that, but this would have an impact on other ranges as well, and 100% input RH is more the exception than the rule. Most hygromethers can't even measure values that high. And the measurement error of a hygrometer is typically much higher than the temperature error we get here.

[MarkCiliaVincenti]

I don't know, my BME280s infallibly go to 100% whenever it's raining. The thing is, the first time I introduced the dew point calculations in the code, they used to give the same result as I used to observe this (that's how I realised the dew point is the same as the temperature at 100% RH), and now somehow that's changed now, and it's looking like a bug. I think it's worth investigating and comparing with the first version to include the dew point calculation code.

[pgrawehr]

The formula you used in your original PR is quite different from the one we finally used, so small differences are to be expected. I don't think we have a bug here, since the different formulas available for the dew point just have their pros and cons - which I can't judge for "better" or "worse" at this time.

When your BME reads 100% when it's raining, I think there's something wrong. This should only happen in very dense fog, unless the sensor gets soaked, which shouldn't happen.

[MarkCiliaVincenti]

... Or, you know, you live on a small island (Malta) where 100% humidity is quite common even without rain :(

I will investigate the formulas a bit better, I think we can also improve accuracy by using the barometric air pressure in the formula. https://en.wikipedia.org/wiki/Dew_point#Calculating_the_dew_point

[pgrawehr]

Would be great if we can improve that. The reason I swapped the original formula you proposed was just because I had to swap it for one that was certainly free to use. Whether the implementation is fully correct and appropriate for all cases I cannot tell (and apparently failed to verify).

[MarkCiliaVincenti]

Well I gave it a good try, but I'm still clueless following https://en.wikipedia.org/wiki/Dew_point#Calculating_the_dew_point and https://en.wikipedia.org/wiki/Arden_Buck_equation

Here's where I am stuck:

1) The 3rd set of formulas on https://en.wikipedia.org/wiki/Dew_point#Calculating_the_dew_point (the part that starts "For greater accuracy") and I understand how they got Psm(T), which is the Arden Buck (https://en.wikipedia.org/wiki/Arden_Buck_equation) equation but specifically for temperatures > 0C. First question: should we change the saturated vapor pressure calculations to use the Arden Buck full equation? That is using two sets of numbers for a,b,c and d? Our functions CalculateSaturatedVaporPressureOverWater and CalculateSaturatedVaporPressureOverIce are using calculations from 1982 but Buck (1996) suggests different numbers.

2) If the temperature is <= 0, and we decide to go with the alternate numbers from Arden Buck for the saturated vapor pressure calculation, should we also use these a,b,c,d values for the rest of the dew point calculation?

3) Either I'm blind or I'm not understanding the math, but I can't understand why Ps,m(T) is being calculated when it's then not used in the two lines below it leading to the dew point temperature.

4) Our function CalculateActualVaporPressure is calling CalculateSaturatedVaporPressureOverWater. Should it be calling CalculateSaturatedVaporPressureOverIce when temperature.DegreesCelsius <= 0? This may or may not be related to (2) above, but I'm not sure.

So you can see, I'm a bit lost :)

[pgrawehr]

I'll investigate when I find time, but just these 2ct for now:

Well I gave it a good try, but I'm still clueless following en.wikipedia.org/wiki/Dew_point#Calculating_the_dew_point and en.wikipedia.org/wiki/Arden_Buck_equation

Here's where I am stuck:

1. The 3rd set of formulas on [en.wikipedia.org/wiki/Dew_point#Calculating_the_dew_point](https://en.wikipedia.org/wiki/Dew_point#Calculating_the_dew_point) (the part that starts "For greater accuracy")  and I understand how they got Psm(T), which is the Arden Buck ([en.wikipedia.org/wiki/Arden_Buck_equation](https://en.wikipedia.org/wiki/Arden_Buck_equation)) equation but specifically for temperatures > 0C. First question: should we change the saturated vapor pressure calculations to use the Arden Buck full equation? That is using two sets of numbers for a,b,c and d? Our functions CalculateSaturatedVaporPressureOverWater and CalculateSaturatedVaporPressureOverIce are using calculations from 1982 but Buck (1996) suggests different numbers.

We can test that. Using more complex equations is not a problem as long as you don't have to do them by hand ;-)

2. If the temperature is <= 0, and we decide to go with the alternate numbers from Arden Buck for the saturated vapor pressure calculation, should we also use these a,b,c,d values for the rest of the dew point calculation?

I think we should use numbers that are published together, otherwise we match no known model.

3. Either I'm blind or I'm not understanding the math, but I can't understand why Ps,m(T) is being calculated when it's then not used in the two lines below it leading to the dew point temperature.

This is lacking a link. The german article explains that the first formula combined with the magnus formula returns the desired formula. I cannot quite follow that logic too, but it makes some sense. It also says (formula 15 on https://de.wikipedia.org/wiki/Taupunkt) that all we need is this formula, which even doesn't need the vapor saturation to be computed explicitly:

4. Our function CalculateActualVaporPressure is calling CalculateSaturatedVaporPressureOverWater. Should it be calling CalculateSaturatedVaporPressureOverIce when temperature.DegreesCelsius <= 0? This may or may not be related to (2) above, but I'm not sure.

No, I don't think that would be a good idea. Water doesn't freeze just because the air is <0°, and also the ground is not necessarily frozen then. Especially for calculating the dew point over sea water (i.e. to estimate the probability of fog) this would give clearly wrong values. Sea water doesn't freeze at 0°, in fact most parts of the sea newer freeze, regardless of temperature.

[MarkCiliaVincenti]

I'll investigate when I find time, but just these 2ct for now:

Well I gave it a good try, but I'm still clueless following en.wikipedia.org/wiki/Dew_point#Calculating_the_dew_point and en.wikipedia.org/wiki/Arden_Buck_equation Here's where I am stuck:

1. The 3rd set of formulas on [en.wikipedia.org/wiki/Dew_point#Calculating_the_dew_point](https://en.wikipedia.org/wiki/Dew_point#Calculating_the_dew_point) (the part that starts "For greater accuracy")  and I understand how they got Psm(T), which is the Arden Buck ([en.wikipedia.org/wiki/Arden_Buck_equation](https://en.wikipedia.org/wiki/Arden_Buck_equation)) equation but specifically for temperatures > 0C. First question: should we change the saturated vapor pressure calculations to use the Arden Buck full equation? That is using two sets of numbers for a,b,c and d? Our functions CalculateSaturatedVaporPressureOverWater and CalculateSaturatedVaporPressureOverIce are using calculations from 1982 but Buck (1996) suggests different numbers.

We can test that. Using more complex equations is not a problem as long as you don't have to do them by hand ;-)

Test how? They'll give slightly different values, then what?

2. If the temperature is <= 0, and we decide to go with the alternate numbers from Arden Buck for the saturated vapor pressure calculation, should we also use these a,b,c,d values for the rest of the dew point calculation?

I think we should use numbers that are published together, otherwise we match no known model.

I agree, so that's a 'yes'?

3. Either I'm blind or I'm not understanding the math, but I can't understand why Ps,m(T) is being calculated when it's then not used in the two lines below it leading to the dew point temperature.

This is lacking a link. The german article explains that the first formula combined with the magnus formula returns the desired formula. I cannot quite follow that logic too, but it makes some sense. It also says (formula 15 on https://de.wikipedia.org/wiki/Taupunkt) that all we need is this formula, which even doesn't need the vapor saturation to be computed explicitly:

I'm sorry but my eyes hurt when I see this :) And not sure I understood you either.

4. Our function CalculateActualVaporPressure is calling CalculateSaturatedVaporPressureOverWater. Should it be calling CalculateSaturatedVaporPressureOverIce when temperature.DegreesCelsius <= 0? This may or may not be related to (2) above, but I'm not sure.

No, I don't think that would be a good idea. Water doesn't freeze just because the air is <0°, and also the ground is not necessarily frozen then. Especially for calculating the dew point over sea water (i.e. to estimate the probability of fog) this would give clearly wrong values. Sea water doesn't freeze at 0°, in fact most parts of the sea newer freeze, regardless of temperature.

Ah that's why they use the terminology 'over water' and 'over ice'? Maybe we should put in both sets of numbers then and allow the developer to choose whether it's over water or over ice, and default to 'over water'. Also, I don't quite understand the physics of it. Say you're in -20C by an empty swimming pool. if you put a few centimeters of water and wait a bit it would freeze and become ice. At that point what happens to the air around the swimming pool that warrants using different values for calculating the dew point just because there is ice a few metres away?

[pgrawehr]

I have some ideas why this might be so, but I'll try to get some literature on it if possible.

[MarkCiliaVincenti]

Any news on this please?

[pgrawehr]

@MarkCiliaVincenti : I was able to grab a book on meteorology from the library, but I was not able to really read it yet. It's still on my to-do list.

[pgrawehr]

So finally trying to resolve this.

My book has another formula for the dew point:

double pa = CalculateActualVaporPressure(airTemperature, relativeHumidity).Millibars;
double dewPoint = (423.86 - (234.175 * Math.Log(pa))) / (Math.Log(pa) - 18.89);
return Temperature.FromDegreesCelsius(dewPoint);

The result of the formula is within 0.2 of the previous one for the test cases. When calculating with 100% RH, it's results are slightly better than the previous formula for temperatures above 0°C, but below the error is also around 0.2°C. It does seem that there is not the correct formula, and every scientist (or meteorological institute) uses slightly different calculations. Since the measurement accuracy for many sensors (especially humidity) is not better than that, I guess it is not possible or useful to find more precise calculations.

By the way: My book uses even three different sets of constants to calculate the saturation pressure. One set for water temperatures 0°-100°C, one set for Water < 0° and one for ice (which is not the same). The reason why the saturation pressure over ice is lower than over water is because the molecules in ice are less likely to be able to flee from their grid than those in water. Therefore, fewer molecules try to enter a volume of air above the surface. In turn, fog is (slightly) less likey to start condensing over ice than over water with the same temperature.

[MarkCiliaVincenti]

Hi,

Yes we already discussed re saturation pressure over ice back in November in this thread. I think the original formula we had that had to be replaced due to being unable to found sources for it, was probably the most accurate.

[pgrawehr]

The question I have is what "most accurate" here means. All these formulas seem to be created from experience, there's no unique definition of them, nor is there a "right" or "wrong". That particular formula was probably most correct near 100%, but we do not know how it behaved in other situations.

Meteorology is not an exact science, since there are always more factors involved than anyone can measure at a time. In this particular case, the dew point is the temperature where - all other parameters staying equal - fog would start building up. I doubt that you can exactly prove this under lab conditions, let alone in nature.

[Ellerbach]

[Triage] My understanding in this thread is that there are multiple formulas used in the literature. It seems there is no perfect one, some better for some usage, some others for other usage. I feel that @pgrawehr's one is working well for most cases. And my preference would be to stick with what is currently implemented. If like @MarkCiliaVincenti anyone wants to implement his own formula, in an extension for example, this seems to be fairly easy and straight forward. @pgrawehr and @MarkCiliaVincenti do you agree with my proposed approach?

[pgrawehr]

Closing as per above