Adding FoV detection, improved groundscatter flag

[aburrell]

I would like to add my automated field of view (FoV) detection and improved groundscatter flagging to RST. This won’t be simple, since we’ll need an “after fitacf, before grd” file as output that the make_grd routines can read. Would anyone be interested in helping this development and/or determining what these new files should look like and be called?

references: http://dx.doi.org/10.1002/2015RS005808, and another RS paper that's still under review or any of my conference presentations in the last year.

[kkotyk]

Can this be prototyped in python first? The python tools I started would make this way easier to start with since they can basically write any python dictionary to dmap.

[kkotyk]

If we have something finalized it would make it much more easy to translate this into RST.

[aburrell]

I have the fov code in davitpy already. The Ribiero-like reflagging doesn't work well in python (thanks to the depth-first search), but I have C code that works and has run on about 4 years of data so far. Right now this all works by outputting ascii files.

[kkotyk]

Sounds good. Well I definitely think we should figure out the scope of work and decide on the best of course of action from there! @egthomas will definitely know more about judging the scope of work of adding a feature to make_grid.

[egthomas]

@aburrell - so these will basically be files that contain all of the same information as a normal fit/fitacf file, but also some additional information about front/rear FOV and a more accurate ground scatter determination?

From working with the map files, we found it is not very difficult to add new data fields to dmap-format files. Maybe we could do something similar by adding additional data fields to the fit/fitacf format files? For example, upon creation with make_fit those fields would be empty, but after running your post-processing software those fields would be populated? There is already some precedent for this with the xtd flag in grid files which indicates whether power and spectral width information is stored in addition to the gridded velocity values.

An alternative would be to take the cfit approach where you create an entirely new file format and introduce another if/else branch within each binary for loading data from a particular file format. I would prefer to avoid this option as it would significantly increase the amount of new code to be written.

[egthomas]

Sorry, we wouldn't be adding new fields to fit-format files, only the fitacf (dmap) format files.

[aburrell]

I prefer the first option as well. From what I remember we'd need to add a FoV flag and decide whether or not to keep the old elevation angle. The groundscatter flag would have a new possible value, but the old values would mean the same thing and existing code should not be affected.

[aburrell]

Another thing to keep in mind is whether we want this code to be operational in real-time or not. I know how I'd like to structure the processing, but haven't needed it and so haven't mocked it up.

[egthomas]

When you say operational in real-time, do you mean at the radar site? Or something that can be located off-site and takes the real-time fitacf-type data stream as input (ie http://superdarn.ca/real-time-display)?

[aburrell]

Something that can be located off-site and take real-time fitacf-type data as input.

[kkotyk]

Do you see this being valuable for the realtime display or do you have some other realtime process you'd like to run?

[aburrell]

When I presented this algorithm at the Royal Astronomical Society special meeting (2016), one of the questions was whether it was possible to run in real time or not. So, I assume there is some interest in having this available. I don't need it yet; I'm still stuck in the 23rd solar cycle.

[egthomas]

If the underlying algorithm is implemented in an RST library then it should be fairly straightforward to write top-level binaries which can handle either a standard fitacf file or the real-time fitacf data stream as input.

[pasha-ponomarenko]

Apologies for trailing behind -- I am in Australia right now... @aburrell, it would be great to implement both your algorithms. Adding an extra field to the already exisitng files seems to be logical. I was thinking about this while considering implementetiaon of the two-component fitting for separating ground and ionospheric scatter in the mixed ACFs www.ann-geophys.net/26/1517/2008/ . Your algorithms require accurate estmates of elevation which is currently not always the case, so there should be some sort of elevation quality indicator in order for the code to decide whether to run these algorithms or not.

[aburrell]

An elevation quality indicator is a good idea. However, you can run both of these methods without calibrated elevation angles. In both cases, the results are usable but not quite as good:

1) The FoV detection suffers near aliasing boundaries, but performs well in other areas 2) The ground/ionospheric reflagging can use a modelled virtual height instead of the calculated virtual height. At the high latitude radars (basically those most like SAS), the Chisham model does a good job grouping the data together for assessment.

[aburrell]

Ok, I am actively working on this, and need to implement the MINPACK least-squares fitting routine as a part of one of the subroutines. I have been looking at include/analysis/mpfit.h and codebase/superdarn/src.lib/tk/lmfit.1.0/src/lmfit.c for inspiration, but am having trouble interpreting some of the comments. Has anyone looked at those recently?