Impact of new elevation algorithms

[ksterne]

Question

Question: Maybe I missed the impact of #93 or maybe other pull requests since then, but does at least that one impact all fitacf 2.5 files?

Category

• [X] Usage
• [ ] Troubleshooting
• [ ] Software Design
• [X] Data related
• [ ] Capabilities
• [ ] Clarity
• [X] Workflow

Details

I was testing out #411 and noticed the values for a fitacf 2.5 file originally generated back in 2015 had completely different values than the file generated with the bugfix branch. Not hugely different (as though I'm seeing errors), but none of them were the same. I guess I just wanted clarification that any and all fitacf 2.5 files generated before that PR was merged in (and deployed on local systems) should be re-generated as it wasn't apparent to me the impact to data products. It's possible the impact was announced, I missed it, and that's why I'm asking the question now. If the answer is a simple yes, then that is OK.

Example

From 2015 file:

float   "elv" [24]=
                28.800341,      30.337442,      0.000000,       28.210726,      0.000000,       33.660332,      15.380051,      25.137430,      31.136417,      33.641632,      26.692331,      30.699003,      30.065453,      35.177666,      29.661558,      30.568630,      16.209198,      4.868846,       29.139009,      26.731544,      12.355013,      15.989361,      19.988621,      0.000000

 short   "slist" [24]=
                2,      3,      23,     24,     25,     26,     27,     28,     29,     30,     31,     32,     33,     34,     35,     37,     38,     39,     40,     41,     44,     45,     46,     75

From develop:

        float   "elv" [14]=
                33.496891,      33.647827,      15.341077,      25.116598,      31.121792,      33.629112,      26.673311,      30.683987,      30.049854,      35.166348,      29.645582,      18.424086,      16.286451,      15.074297

short   "slist" [14]=
                24,     26,     27,     28,     29,     30,     31,     32,     33,     34,     35,     38,     44,     45

I'm also noticing just now that the array size is smaller on the new array since the slist is smaller. So, it's a little hard to compare values. Looking a little harder, most of them are closer than I thought, but the value for range gate 24 is 5 degrees different.

Code

make_fit processing

Installation Process

Standard RST compilation between branches on Ubuntu 20.

Documentation

Nothing additional to what is provided.

[ecbland]

@ksterne Which radar(s) did you look at? #93 was merged in 2019, so a lot of fitacf files created before that will have used the old elevation angle calculation. We expect the elevation angles to be different when the interferometer offset has a X or Z component

[ksterne]

My apologies, I think the values I gave are with cvw data. I wasn't asking more about why the values are so different, but more on the impact of data products and needing to reprocess data.

[ecbland]

CVW has a simple interferometer offset, so the elevation angles should not have changed when the new algorithm was introduced. But to answer your question, yes you may want to reprocess the data. The new elevation algorithm has also been added to FITACF3 on the develop branch -- it will be included in the next release.

[pasha-ponomarenko]

@ksterne , the new algoithm should provide a noticeable difference in the data from radars with a considerable ofset in X direction (perpendicular to the boresight) as this offset it not accounted for in the original algorithm except that there was a separate code for Goose Bay which used to have a 15.5-m X-offset. While the X-offset is not accounted for in the original code, there is a patch-up fix for Z-offset. Currently, there are five radars with X-offset: Goose Bay (+1.5 m), INV (+1.5 m). Syowa E&W (-5 m) and ZHS (-27.6 m).

[ecbland]

@ksterne Could this be related to phidiff (see #345)? I don't know the dates for the data you were comparing, but CVW has phidiff=-1 for pre-2013 data.

elevation_v2 deals with phidiff differently (by not using it!)

https://github.com/SuperDARN/rst/blob/97f26d65a45a83cc2f96c808682061911efa905b/codebase/superdarn/src.lib/tk/elevation.1.0/src/elevation_v2.c#L65-L89

[pasha-ponomarenko]

@ecbland, phi0 sign correction with phidiff occurs upstream from the elevation determination (this is duly replicated in FITACF3 as well):

The old elevation code utilises its value only in a term called elev_corr which is used for the elevation correction due to the offset (if any) in Z direction:

As this offset is normally relatively small, the resulting difference in the elevation code output should be small.

Furthermore, I am not sure if phidiff should be applied to elev_correct at all -- this might be yet another undetected algorithmic bug in the original FITACF. I just need to have a closer look.

[mts299]

@ksterne @ecbland @pasha-ponomarenko what is the status of this?

I know we just released Fitacf 3.0 with elevation v2 as default I believe?

[ecbland]

I know we just released Fitacf 3.0 with elevation v2 as default I believe?

Yes, although this issue is about fitacf2.5

I don't think we have enough information to investigate this issue. @ksterne?

[pasha-ponomarenko]

@ecbland and @ksterne , even after looking through the previous comments I am not sure if I clearly understand what is the actual problem here.

[mts299]

Sorry @ecbland my bad

[ksterne]

My question was should any and all (just to be easy/safe) fitacf files processed with fitacf 2.5 be reprocessed due to updates in the code? Again, all here is laziness on not wanting to figure out which radars might have what offset.

This question came up because I was comparing a newly processed fitacf2.5 file with a file processed back in 2015. and the values were different. I'll try to give better examples for what processing of what files leads to what answer in the near future (I'll need to refer to my notes of what exactly I was looking at).

[pasha-ponomarenko]

@ksterne, I would not hurry into the total re-processing as there is an on-going work on getting tdiff right otherwise the re-processing would not make much sense. In general, the majority of the radars are not affected (no X or Z offset) or affected very little (Z offset only) by the changes in the processing code. The most significantly affected ones are those with the offset in X direction (Goose Bay (+1.5 m), INV (+1.5 m). Syowa E&W (-5 m) and ZHS (-27.6 m).), and most seriously -- Zhong Shan, but I don't know if its current tdiff of -180 ns is actually correct.

[mts299]

Status on this PR?

[ecbland]

Closing this issue due to lack of activity.