Stereo offset for legacy data

Buried deep in fitacf.c of FITACF 2.5 there is the following comment:

    /* need to incorporate Sessai's code for setting the offset
       for legacy data here.
    */

which, as far as I can tell, has been present in this file / FITACF since at least 2006 (e.g., https://github.com/SuperDARN/rst-archive/blob/rst.2.05/code/src.lib/superdarn/fitacf.1.22/src/fitacf.c#L169-L171). For context, our software currently assumes the offset value for Stereo data in dat- and fit-format files is stored in the usr_resL1 field.

In a copy of Sessai's code, here was his explanation of this issue:

We should be very careful for the fact that
some RCPs for mono mode use usr_resL1 for some other purposes!!!
also note that there was an period when
the 'CHANNEL' parameter had not been introduced in Radops parms,
in the very early stereo era.
So in order to know if the data is stereo or mono correctly,
we should check all of ST_ID, YEAR, CHANNEL and usr_resL1!!!!!!!

I'll try reaching out to Sessai, but my impression is that this fix needs to be implemented at a more fundamental level (ie at the I/O stage within OldRawReadCurrent of the oldraw library and OldFitReadCurrent of the oldfit library) rather than within a later processing stage such as FITACF. And I suppose the native IDL dat / fit code may require modifications too...

Following up on this, at least one control program (cpid = 4204, from 1998-2009) used the usr_resL1 field to store information other than a Stereo offset value. This will affect FITACF 2.5 such that the wrong Badlags function will be called, eg (https://github.com/SuperDARN/rst/blob/main/codebase/superdarn/src.lib/tk/fitacf.2.5/src/do_fit.c#L195-L196)

  if (iptr->prm.offset==0) FitACFBadlags(&iptr->prm,&badsmp);
  else FitACFBadlagsStereo(&iptr->prm,&badsmp);

although FITACF 3.0 appears to be unaffected since it also compares against the channel number, eg (https://github.com/SuperDARN/rst/blob/main/codebase/superdarn/src.lib/tk/fitacf_v3.0/src/preprocessing.c#L609)

  if((offset != 0) && ((channel == 1) || (channel == 2))) {

As an aside, LMFIT (v1) appears to handle this completely incorrectly (comparing against channel only), eg (https://github.com/SuperDARN/rst/blob/main/codebase/superdarn/src.lib/tk/lmfit.1.0/src/lmfit.c#L392-L397)

  /*calculate noise levels*/
  if(prm->cp != -3310 && prm->cp != 3310 && prm->cp != 503 && prm->cp != -503)
  {
    if(fblk->prm.channel==0) FitACFBadlags(&fblk->prm,&badsmp);
    else FitACFBadlagsStereo(&fblk->prm,&badsmp);
  }

(edit note that FITACFEX2 also uses this incorrect logic)

while LMFIT v2 does not appear to treat Stereo data differently at all (at least based on either the channel or offset fields), which doesn't seem quite right.

Getting back to the early Stereo mode data, it appears the channel field was not introduced to the dat and fit parameter block at HAN/PYK until 2002-04-24. So prior to that Stereo HAN and PYK files have a (correct) non-zero offset but no channel information. As far as Tim and I can tell, the potentially very early Stereo data where neither the offset or channel information are recorded were either never distributed or do not exist.

Also, because STO dat files from Jul 2006 - Jan 2009 were converted to rawacf with dattorawacf and we no longer have the originals, STO rawacf files on the data servers with the 4204 CPID also have a non-zero offset, eg (20090101.0000.11.sto.rawacf):

scalars:
    char    "radar.revision.major" = 1
    char    "radar.revision.minor" = 66
    char    "origin.code" = 1
    string  "origin.time" = "Wed Feb 25 15:55:37 2009"
    string  "origin.command" = "/home/dproc/bin/dattorawacf /data1/tmp/tmp.T30861"
    short   "cp" = -4204
    short   "stid" = 8
    short   "time.yr" = 2009
    short   "time.mo" = 1
    short   "time.dy" = 1
    short   "time.hr" = 0
    short   "time.mt" = 0
    short   "time.sc" = 11
    int "time.us" = 0
    short   "txpow" = 9000
    short   "nave" = 28
    short   "atten" = 0
    short   "lagfr" = 1200
    short   "smsep" = 200
    short   "ercod" = 0
    short   "stat.agc" = 31744
    short   "stat.lopwr" = -8252
    float   "noise.search" = 434.000000
    float   "noise.mean" = 2261.000000
    short   "channel" = 0
    short   "bmnum" = 15
    float   "bmazm" = -34.325001
    short   "scan" = 1
    short   "offset" = 1
    short   "rxrise" = 100
    short   "intt.sc" = 3
    int "intt.us" = 0
    short   "txpl" = 200
    short   "mpinc" = 2400
    short   "mppul" = 7
    short   "mplgs" = 18
    short   "nrang" = 70
    short   "frang" = 180
    short   "rsep" = 30
    short   "xcf" = 0
    short   "tfreq" = 13220
    int "mxpwr" = 4400000
    int "lvmax" = 20000
    int "rawacf.revision.major" = 6
    int "rawacf.revision.minor" = 5
    string  "combf" = "$Id: fte_4204.c,v 1.9 2000/11/29 17:16:37 barnes Exp $"
    float   "thr" = 3.000000

So either these files need to be corrected, or the rawacf reading software will need to be modified too. Or maybe the logic to handle this should go inside all of the fitting routines too?

SuperDARN / rst

Stereo offset for legacy data #580