Transforms that compress an axis such as var[E=@AVE] which no longer depends on E, may be computed by forming the sums and weights in chunks. e.g. E=1:6, E=7:10, then computing the final result. The size of the chunk depends on the memory size setting and on the variable type, i.e. user-variable or file variable. Here is an example creating an example dataset and going through Ryo's steps:
! Define a variable in x-y-e and save to a file
use etopo20
let rosetmp = if rose le 3000 then rose
let ens = randu(_m[m=1:10])
let fvar = rosetmp * ens
save/clobber/file=xye.nc/m=1/mlimits=1:10 fvar
set mem/siz=100
repeat/m=2:10 ( save/append/file=xye.nc fvar)
cancel data/all; cancel var/all; cancel memory/all
! Now open this file and write out averages over E to files, first a
! user-variable and then the file variable. The results should be the same.
use xye.nc
let ufvar = fvar[d=1]
! This memory setting causes ufvar[M=1:10@AVE] to be computed in chunks.
set mem/siz=15
save/clobber/file=tmp-ufvar.nc ufvar[M=1:10@AVE]
show memory/diagnostic
save/clobber/file=tmp-fvar.nc fvar[M=1:10@AVE]
show memory/diagnostic
cancel data/all; cancel var/all; cancel memory/all
! Now look at the difference between these m-averages by plotting them
! averaged over X, they should be identical lines.
use tmp-ufvar.nc
use tmp-fvar.nc
let diff = ufvar[d=1] - fvar[d=2]
plot diff[x=@MAX], diff[x=@MIN]
this was fixed in setup_gather.F -- a slight logic error in the changes when we implemented ability to do the computation in chunks for better memory use.
Ryo Furue reported this. https://www.pmel.noaa.gov/maillists/tmap/ferret_users/fu_2019/msg00456.html
This first appears in Ferret v7.2.
Transforms that compress an axis such as var[E=@AVE] which no longer depends on E, may be computed by forming the sums and weights in chunks. e.g. E=1:6, E=7:10, then computing the final result. The size of the chunk depends on the memory size setting and on the variable type, i.e. user-variable or file variable. Here is an example creating an example dataset and going through Ryo's steps: