NOAA-PMEL / Ferret

The Ferret program from NOAA/PMEL
https://ferret.pmel.noaa.gov/Ferret/
The Unlicense
55 stars 21 forks source link

AVE or DIN returns wrong result for some chunk size along AVERAGE axis #1964

Closed AnsleyManke closed 4 years ago

AnsleyManke commented 4 years ago

Ryo Furue reported this. https://www.pmel.noaa.gov/maillists/tmap/ferret_users/fu_2019/msg00456.html

This first appears in Ferret v7.2.

Transforms that compress an axis such as var[E=@AVE] which no longer depends on E, may be computed by forming the sums and weights in chunks. e.g. E=1:6, E=7:10, then computing the final result. The size of the chunk depends on the memory size setting and on the variable type, i.e. user-variable or file variable. Here is an example creating an example dataset and going through Ryo's steps:

! Define a variable in x-y-e and save to a file
use etopo20
let rosetmp = if rose le 3000 then rose
let ens = randu(_m[m=1:10])
let fvar = rosetmp * ens
save/clobber/file=xye.nc/m=1/mlimits=1:10 fvar
set mem/siz=100
repeat/m=2:10 ( save/append/file=xye.nc fvar)

cancel data/all; cancel var/all; cancel memory/all

! Now open this file and write out averages over E to files, first a 
! user-variable and then the file variable.  The results should be the same.

use xye.nc
let ufvar = fvar[d=1]

! This memory setting causes ufvar[M=1:10@AVE] to be computed in chunks.
set mem/siz=15

save/clobber/file=tmp-ufvar.nc ufvar[M=1:10@AVE]
show memory/diagnostic

save/clobber/file=tmp-fvar.nc  fvar[M=1:10@AVE]
show memory/diagnostic

cancel data/all; cancel var/all; cancel memory/all

! Now look at the difference between these m-averages by plotting them
! averaged over X, they should be identical lines.

use tmp-ufvar.nc
use tmp-fvar.nc

let diff = ufvar[d=1] - fvar[d=2]

plot diff[x=@MAX], diff[x=@MIN]
AnsleyManke commented 4 years ago

Here is the plot made by the example script

example

AnsleyManke commented 4 years ago

this was fixed in setup_gather.F -- a slight logic error in the changes when we implemented ability to do the computation in chunks for better memory use.