Find out which values fail between the ISO 2533 tables and the calculated values

[newbthenewbd]

Assuming #20 gets merged... Awfully many, actually? For whatever reason, the calculations seem to be failing for most of the tests, bar these we originally had... On a quest to find why :)

[newbthenewbd]

Found an error that the upper layer was being selected in a place, rather than lower; besides that, only pointing a finger at it so far, but it seems that the data in rows-H is actually more reliable?

[newbthenewbd]

E.g. calculations from the corrected layer version:

H = -1750.0
h = -1749.5183624606748
TK = 299.525
TC = 26.375
p_mbar = 1241.913519293671
p_mmhg = 931.5117440492897
rho = 1.4444295950403332
g = 9.81205023017224
p_p_n = 1.2256733474400898
rho_rho_n = 1.179126200032925
root_rho_rho_n = 1.08587577559909
a = 346.94569751242653
mu = 1.8437529742282412e-05
v = 1.2764574892116896e-05
lambda = 0.026232628838116796
H_p = 8762.644925568891
gamma = 14.172815740483097
n = 3.0034015383879232e+25
v_bar = 467.9155869020716
omega = 8318241862.2816305
l = 5.625174530861431e-08

Data from rows-h, rows-H:

- H: -1750.0
  h: -1750.0
  TK: 299.528
  TC: 26.378
  p_mbar: 1.24198e+3
  p_mmhg: 9.31563e+2
  rho: 1.44449e+0
  g: 9.8121
  p_p_n: 1.22574e+0
  rho_rho_n: 1.17918e+0
  root_rho_rho_n: 1.08590e+0
  a: 346.948
  mu: 1.8438e-5
  v: 1.2764e-5
  lambda: 2.6233e-2
  H_p: 8762.7
  gamma: 1.4173e+1
  n: 3.0035e+25
  v_bar: 467.92
  omega: 8.3187e+9
  l: 5.6249e-8

- H: -1750.0
  h: -1750.0
  TK: 299.525
  TC: 26.375
  p_mbar: 1.24191e+3
  p_mmhg: 9.31512e+2
  rho: 1.44443e+0
  g: 9.8121
  p_p_n: 1.22567e+0
  rho_rho_n: 1.17913e+0
  root_rho_rho_n: 1.08588e+0
  a: 346.946
  mu: 1.8438e-5
  v: 1.2765e-5
  lambda: 2.6233e-2
  H_p: 8762.6
  gamma: 1.4173e+1
  n: 3.0034e+25
  v_bar: 467.92
  omega: 8.3182e+9
  l: 5.6252e-8

Another data point, calculations:

H = 700.0
h = 700.0770917105015
TK = 283.59999999999997
TC = 10.449999999999989
p_mbar = 931.9355588813422
p_mmhg = 699.0091534625044
rho = 1.1447687266855282
g = 9.80449032414095
p_p_n = 0.9197488861399873
rho_rho_n = 0.9345050830085944
root_rho_rho_n = 0.9666980309324077
a = 337.596613121409
mu = 1.7673421772376575e-05
v = 1.5438421194075406e-05
lambda = 0.024984116096128255
H_p = 8303.154212515834
gamma = 11.223873904167418
n = 2.3803168853859527e+25
v_bar = 455.3067482821222
omega = 6414894187.583608
l = 7.097650713405089e-08

Data from rows-h, rows-H:

- H: 700.0
  h: 700.0
  TK: 283.601
  TC: 10.451
  p_mbar: 9.31944e+2
  p_mmhg: 6.99016e+2
  rho: 1.14478e+0
  g: 9.8045
  p_p_n: 9.19757e-1
  rho_rho_n: 9.34512e-1
  root_rho_rho_n: 9.66702e-1
  a: 337.597
  mu: 1.7673e-5
  v: 1.5438e-5
  lambda: 2.4984e-2
  H_p: 8303.2
  gamma: 1.1224e+1
  n: 2.3803e+25
  v_bar: 455.31
  omega: 6.4149e+9
  l: 7.0976e-8

- H: 700.0
  h: 700.0
  TK: 283.6
  TC: 10.45
  p_mbar: 9.31935e+2
  p_mmhg: 6.99009e+2
  rho: 1.14477e+0
  g: 9.8045
  p_p_n: 9.19748e-1
  rho_rho_n: 9.34505e-1
  root_rho_rho_n: 9.66698e-1
  a: 337.597
  mu: 1.7673e-5
  v: 1.5438e-5
  lambda: 2.4984e-2
  H_p: 8303.2
  gamma: 1.1224e+1
  n: 2.3803e+25
  v_bar: 455.31
  omega: 6.4149e+9
  l: 7.0977e-8

[newbthenewbd]

@ronaldtse Should we only use the rows-H data here, or any insights on what's going on?

[newbthenewbd]

Thinking about it for a while more, I guess that rows-h would actually need to calculate based on the geometric altitude for us to get the correct results from it...

[newbthenewbd]

With the latest version in #20, seems like the constant thing for many variables is actually the number of significant digits, not decimal places, gotta adjust accordingly...

[ronaldtse]

I can see there is now a pattern:

20321 examples, 2830 failures

Among the failures:

• lambda: 1 failure
• g: 1
• mu: 2
• v_bar: 3
• h: 7
• a: 9
• H_p: 9
• n: 43
• omega: 46
• gamma: 48
• l: 57
• v: 59
• p_mbar: 439
• p_p_n: 441
• rho: 454
• rho_rho_n: 463
• p_mmhg: 492

[ronaldtse]

Clearly, g, mu, h, H_p have to be correct for others derived values to be correct... So starting with the least failures working onto the rest.

[newbthenewbd]

I just wonder if you might be authorized to give me (in a private channel, I suppose) the whole tables to look at?

Forty minutes (and some slower, very precise calculators) in, if some of these are not OCR errors, I'm thinking that some imprecisions might actually have to be introduced this time to replicate some of these results? Now going to look at a Minsk-22 emulator, meanwhile :^)

Edit: the emulator idea doesn't look all that promising after all...

[ronaldtse]

@newbthenewbd the full document(s) can be found at https://github.com/metanorma/iso-2533/ where you have access.

The tables are already encoded in YAML as the files you originally processed.

[ronaldtse]

@newbthenewbd if we are seeing rounding / floating point issues we need to report them back to the committee that holds this document.

For better accuracy, we might want to use the advanced number handling functionality in Ruby such as BigDecimal.

[newbthenewbd]

@newbthenewbd the full document(s) can be found at https://github.com/metanorma/iso-2533/ where you have access.

Oo, thank you! But it seems that I don't actually have access to this repo... The YAML version has clearly been OCR-ed and contains at least some errors, e.g. from my git diff right now:

--- a/spec/fixtures/iso-2533-1975-table5.yaml
+++ b/spec/fixtures/iso-2533-1975-table5.yaml
@@ -9823,7 +9823,7 @@ rows-H:
     temperature-K: 277100
     temperature-C: 3950
     p-mbar: 8.25013e+2
-    p-mmHg: 6.l8811e+2
+    p-mmHg: 6.18811e+2
     density: 1.03720e+0
     acceleration: 9.8014

Do you mean that these should be reported in a special way?

I know about BigDecimal, the thing is, for example 6356766×30650/(6356766−30650) pretty sure just really is < 30798.5, not >=, as would be needed for this to pass:

1) Atmospheric::Isa variable (h) outputs conform to test values variable (h) @30650.0 outputs conforms to test value
     Failure/Error: expect(calc).to eq(expected_value)

       expected: 30799.0
            got: 30798

       (compared using ==)
     # ./spec/atmospheric/isa_spec.rb:58:in `block (5 levels) in <top (required)>'

[newbthenewbd]

I am basically not saying that the tables are per se incorrect (well, except for the YAML ones, that's on the OCR :^) ), rather, the exact process through which they were devised had been lost along with a 37-bit computer from the 70's, so once improving accuracy doesn't help... I really hope the OCR was just really bad, and would like to check :))

[ronaldtse]

@newbthenewbd Let me fix the access permissions. We obviously want to fix the OCR errors.

Before we report back to the committee we want to be sure that the values we report to be "not reproducible" are really non-reproducible even when we use a more "modern" accuracy. 😉

[ronaldtse]

@newbthenewbd fixed!

[newbthenewbd]

Hacked together a quick-and-terrible BigDecimal version, all the errors are still here... Feeling guilty working with little to show for it :(

But if I didn't miss something to BigDecimal-ize, I still contend that too low accuracy is less likely to be our culprit than too high, as Ruby uses double precision IEEE 754 floating point numbers (11 bits of exponent, 53 bits of mantissa), and Minsk-22 has, according to the unofficial emulator's documentation:

    Fixed-point numbers: sign and 36 significant bits

    Binary floating-point numbers (from top to bottom bit):

        sign of mantissa
        28 bits of mantissa
        1 bit unused
        sign of exponent
        6 bits of exponent

    Decimal floating-point numbers (used only for printing):

        sign of mantissa
        28 bits of mantissa (7 decimal digits in BCD)
        1 bit unused
        sign of exponent
        1 bit unused
        5 bits of exponent (2 decimal digits in BCD, the former one truncated to 1 bit)

So, just with regular Ruby floats (equivalent to C doubles), we should actually be more accurate!

[newbthenewbd]

With single-precision floats and the operations reordered like this:

float f = 6356766.0f;
f /= 6356766.0f - 30650.0f;
f *= 30650.0f;
printf("%f\n", f);

We actually turn up the "wanted" result for this one:

30798.500000

But the problem is, Ruby doesn't have these!

[ronaldtse]

@newbthenewbd is it possible to use a "diff in least significant digit" difference indicating that the number we get is nearly identical to the original except for the least significant digit?

ie. if a failure is a difference in the last significant digit, we consider that a float implementation difference, not a real error.

[newbthenewbd]

That could work as a last resort; I also want to look a little bit closer at these variables with many more failures, could be something more easily correctable there, just haven't gotten to it yet :)

[ronaldtse]

@newbthenewbd once this is somewhat done (matching the numbers isn't the most important) we want to move onto the other tables. Thanks!

[newbthenewbd]

As in, the extra formulas, or table generation? :)

[ronaldtse]

Both please 😊

[newbthenewbd]

Eventually of course, haha :)

[newbthenewbd]

Ok, getting to it when I get home today, definitely waited long enough already... :)

[newbthenewbd]

Found something interesting, apparently the original process did limit the pressure layers to six significant digits right in the calculations, that brings it down to 1847 errors... Taking a break again soon, but this time only for a couple hours :smile:

[newbthenewbd]

Or not, because an experiment to try to get some of the calculations closer to their 32-bit float results at least (ugh, Ruby) makes for only 1816 errors...

Honestly, I believe that it might take doing this in a more powerful language (with more powerful tools already existing), or inventing some very arcane Ruby ourselves, to match them all :( Going to push this unfinished experiment to a separate f32 branch, presuming that the version sans my attempts is actually closest to truth, and do as you say!

[ronaldtse]

@newbthenewbd in any case the point is not really to "match" the numbers, but to say that our numbers are "okay" with a defined rule, e.g. in a range of the (second least significant digit). Once we can make that claim we're already good.

[ronaldtse]

@newbthenewbd so we currently have 401 values out of 20321 calculations that deviate in the last significant digit from the original tables due to floating point rounding:

https://github.com/metanorma/atmospheric/actions/runs/8487801016/job/23256038850

According to the values, these differ in the last significant digit by 2.

Can we have a "relaxed" mode that we relax the rules so that they pass?