Noisy streaks in range related to lag0 power?

[egthomas]

I'll probably expose my ignorance of the workings of fitacf here, but I thought it would be more useful to raise this point in a new issue rather than continue in the recently closed pull request. I've repeated the comparison of fitacf data genereted using fitacf.2.4 (rst.3.2), fitacf.2.5 (rst.3.6), and fitacf.3.0 for a new date (20161029) and again for CVE.

This first attached figure shows the velocity data for Beam 2. Immediately the vertical streaks in the fitacf.3.0 data between 08:00-11:00 UT jumped out at me:

Ok, that's weird, next I checked the power:

Nothing out of the ordinary here (and it is a little more obvious how more scatter is available with fitacf.3.0 vs fitacf.2.5). Eventually I checked the lag0 power:

Could this be the culprit? There is also still a fair amount of messy looking velocity data with fitacf.3.0 between ranges 40-50 from 16:00-24:00 UT (near the far edge of the ground scatter band).

[pasha-ponomarenko]

Thanks a lot, Evan! This is what we need: to find what might be wrong with FITACF3. How frequently do you see these streaks? They look like some sort of a pulsed inerference.

I heave already mentioned my concern about the "salt-and-pepper" velocity points. I already spent some time trying to identify their exact cause -- it is not limited to low snr, there is something else in play as well.

[pasha-ponomarenko]

Right now I am away form the office taking a well-deserved three-day break at the seaside, where the internet is not too fast so my analysis is limited to some Hokkaido East data that happened to be on my comp. After looking at the data for a while I have a feeling that the source for the "messy" velocity values lies most probably with the phase-unwrapping part of the code. For some unknown reason it sporadically shifts low-power ground scatter velocity in a step-like manner up or down by an integer multiple of some quantisation constant. In HOK case this constant is ~450 m/s producing additional clusters of velocity values at +-450 m/s, +-900 m/s etc. Tomorrow I will be back to the town and should be able to have a closer look. Testing this hypothesis by rolling back to the FITACF2 version of the "unwrapping" algorithm might be feasible, but I suspect that the latter is so firmly entrenched into the phase fitting routine(s) that it might be not easy to extract it from there... At the same time I should be able to look closer at the streaky part of the CVE data above.

At the end of the next week we are flying back to Canada, so I might be busy with wrapping things up...

[egthomas]

Don't let me keep you from enjoying your vacation! I haven't had a chance to look at many other days yet so I can't say whether this is a recurring feature. Right now I have our compute servers focused on processing grid data - soon I should be able to redirect them to processing rawacf data with fitacf3.

[pasha-ponomarenko]

OK, Evan, in a meantime I will try to figure out how to deal with the "messy" velocity jumps in low-power ACFs.

Keith, in that respect, how can I extract information on sigma (alpha) from FITACF3? I remember that it was possible to print some intermediate parameters into a test ASCII file...

[kkotyk]

@pasha-ponomarenko In fitacftoplevel.c youll notice there are commented out statements to print range nodes. you can either uncomment them, or place that statement after the function youd like to see output after. Its placed in a file called fullrangeinfo.txt. Make sure you delete that file between successive runs.

[pasha-ponomarenko]

Thanks Keith! I think I managed to figure out the problem and find some sort of an acceptable solution. Now, can you help me with implementing the changes? In preprocessing.c, I am adding several lines to ACF_Phase_Unwrap routine, which require using fit_prms->mpinc and fit_prms->nave. To make these parameters available, I modified the definition of ACF_Phase_Unwrap in preprocessing.h from void ACF_Phase_Unwrap(llist_node range); to void ACF_Phase_Unwrap(llist_node range,FITPRMS *fit_prms); and changed the header of the routine itself in preprocessing.c in the same way. However, this modification produces a segmentation fault.
I made sure that the code works properly if I replace the variables with just numbers (e.g., use sqrt(27) instead of sqrt(fit_prms->nave)). I did not want to mess up with pull requests etc so I just attach the routine as a txt file below ACF_Phase_Unwrap_modified.txt .

[pasha-ponomarenko]

I figured out what was causing the "messy" velocities on the farther edge of the ground scatter band: they are related to false 2Pi shifts during the phase "unwraping", which are caused by excessive phase flucutations in low-power data. My initial fix above was not perfect becasue it was distorting the true high-velocity data (sorry, Keith!) but the latest one aided by fitting error analysis seems to do a good job. Here are the original data and here are the corrected ones I will try to write a comprehensive descritpion of the required mofdification, but I don't promise to do it before coming back to Saskatoon this coming weekend.

The vertical stirpes seem to be caused by some sort of interference so I need to know how frequent this thing is before trying to fix it. I cannot recall seen these in data from other radars...

[kkotyk]

I didn't have time to work on the fix yet so I guess that worked out. I think I will wait until you are back to implement it.

[egthomas]

I'm not sure about other radars but we see these vertical streaks fairly often at CVE. Here's another example from 20 September 2016.

velocity:

lag0 power:

[pasha-ponomarenko]

Did you notice that the lag 0 power of the streaks shows a disontinuity at gate 65?

[egthomas]

Yes, that is odd. I'm also not necessarily convinced that all of the velocity streaks are related to lag0 power, but that's my best guess for the moment. Hope your return from Japan went smoothly.

[sshepherd]

Hi guys, I was talking to Evan about this issue this morning and it reminded me of a few things. I believe the discontinuity is due to the use of the alternate lags for lag0 power at the farther ranges. At ranges beyond the 'critical' range (where the signal from the first pulse arrives during the second pulse) we use the last (I think) pulse to compute the lag0 power. The reason is that the noise adds coherently for lag0 and any signal from pulse two would contribute. I cannot recall how we deal with the signals from previous pulses when using the alternate lags.

[pasha-ponomarenko]

So, we seem to fix the "salt-and-pepper" issue, but the streaks remain there. In order to be able to fix them we need to know what they are or at least what features can be used to filter them out. Any ideas? They do not look like interference from CW because the latter would generate an effectively constant power level at all ranges while the observed signals show significant change with range.

[sshepherd]

I have no access at the moment, but what do the ACFs look like for these times?

On Fri, Apr 7, 2017 at 5:20 PM pasha-ponomarenko notifications@github.com wrote:

So, we seem to fix the "salt-and-pepper" issue, but the streaks remain there. In order to be able to fiix them we need to know what they are or et least what features can be used to filter them out. Any ideas? They do not look like interference from CW because the latter would generate an effectively constant power level at all ranges while the observed signals show significant change with range.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/SuperDARNCanada/fitacf.3.0/issues/7#issuecomment-292654688, or mute the thread https://github.com/notifications/unsubscribe-auth/AC2DPwgfbVPVORwXmSE24Gt1-diixJ1Aks5rtqiQgaJpZM4Ml1Gm .

[pasha-ponomarenko]

Simon, ACFs look funny. Both power and phase lags vs range show some structures, but at this stage it is hard to make sense out of them. More importantly, for each of the streaks velocity vs range shows several saw-tooth flips which indicate jupming through the maximum measured velocity, and between the flips velocity itself increases with range. This should allow to filter them out one way or another.

Furthermore, these things seem to occur when the radar works at 10.4 MHz and are virtually absent at 14.7 MHz. The jump at gate 65, as you mentioned before, is related to the fact that from this range the lag0 power is calculated from pulse 8 rather than pulse 0, and they are separated by 43*mpinc=64.5 ms. So it looks like there is a some sort of CW interference whose duration is comparable to the lengths of a single sampling interval, ~100 ms.

Again, I don't know if this effect is observed by other radars.

[egthomas]

Unfortunately no one seems to bother creating quick-look RTI plots of lag0 power, and these streaks won't show up in the velocity plots on the VT website either because they're using fitacf.2.5. I'll try going through some other days and radars today to see if it occurs anywhere other than Christmas Valley.

[pasha-ponomarenko]

I agree with you, Evan, it should be an option on the VT on-line tool. I will try to look here. In the meantime, I found a way to detect most of them in your data by "unwrapping" the velocity-vs-range dependence: if the maximum "unwrapped" velocity exceeds the maximum "wrapped" one, then there is a high probability that this beam is contaminated by the "streaks".

[egthomas]

I contacted Mike to see if he's ok with me adding it to their web tool or if he would prefer one of their current grad students to do so.

On Simon's data browser he plots lag0 power on a log scale and you can see that even though there is a discontinuity at range gate 65 there is usually something present at all ranges:

edit - sorry, this is 20170108 CVE, beam 3 (http://sdnet.thayer.dartmouth.edu/web_data/2017/01/08/20170108.cve.b03.lg0.ts.html#DATA)

[sshepherd]

Evan,

I have some code that I could point you to to make the lag0 power RTIs, if you're interested.

Headed to the championship game in a few minutes. Sadly we weren't even close to making it...

Simon

On Mon, Apr 10, 2017 at 8:31 AM, Evan Thomas notifications@github.com wrote:

Unfortunately no one seems to bother creating quick-look RTI plots of lag0 power, and these streaks won't show up in the velocity plots on the VT website either because they're using fitacf.2.5. I'll try going through some other days and radars today to see if it occurs anywhere other than Christmas Valley.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SuperDARNCanada/fitacf.3.0/issues/7#issuecomment-292935476, or mute the thread https://github.com/notifications/unsubscribe-auth/AC2DP85UnLdH-uc3jW1AG_HmOA5ohI2rks5ruiEUgaJpZM4Ml1Gm .

[pasha-ponomarenko]

After looking for a while at some other data I would say that these are probably HF signatures of lightning strikes, which are considered to be the main source of the background noise in our data. In your initial example they are just a bit too powerful so they dominate the ionopheric echoes. From what I know, these things last from tens to hundreds milliseconds, which matches well what we see in the data. Are you aware of any regular lightning activity within, say, 1000 km from CVE? Florida?

[sshepherd]

We could look into it when I get back. If that is the source then we should see them at many of the other radars

On Mon, Apr 10, 2017 at 11:00 AM pasha-ponomarenko notifications@github.com wrote:

After looking for a while at some other data I would say that these are probably HF signatures of lightning strokes, which are conidered to be the main source of the background noise in our data. In your initial example they are just a bit too powerful so they dominate the ionopheric echoes. From what I know, these things last from tens to hundreds milliseconds, which matches well what we see in the data. Are you aware of any regular lightning activity within, say, 1000 km from CVE? Florida?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SuperDARNCanada/fitacf.3.0/issues/7#issuecomment-292976307, or mute the thread https://github.com/notifications/unsubscribe-auth/AC2DP1rbmoT-ZaV_nRPyj3WqrKKX7bIHks5rukQUgaJpZM4Ml1Gm .

[egthomas]

I added lag0 power as an option to the VT page with Mike's blessing. I do have my own code to do this but was feeling lazy - sorry Simon!

Fort Hays West also observes these streaks around the same general time; CVE just appears to be much more sensitive (and I happened to choose a camping beam for FHW but not CVE):

[sshepherd]

Is the sky noise jumping at the same time in cve? I don't see the same in fhw.

On Mon, Apr 10, 2017 at 2:54 PM Evan Thomas notifications@github.com wrote:

I added lag0 power as an option to the VT page with Mike's blessing. I do have my own code to do this but was feeling lazy - sorry Simon!

Fort Hays West also observes these streaks around the same general time; CVE just appears to be much more sensitive (and I happened to choose a camping beam for FHW but not CVE):

[image: rti_1491848745] https://cloud.githubusercontent.com/assets/1869073/24876940/878be834-1dfb-11e7-9886-c38344b79316.jpg

[image: rti_1491848706] https://cloud.githubusercontent.com/assets/1869073/24876930/7d4f8eca-1dfb-11e7-91d4-dfd4966d25a9.jpg

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SuperDARNCanada/fitacf.3.0/issues/7#issuecomment-293044715, or mute the thread https://github.com/notifications/unsubscribe-auth/AC2DP2ZMkb4snsXvpo3MbWpuVavaK1xSks5runrSgaJpZM4Ml1Gm .

[pasha-ponomarenko]

This is not surprising because the noise is calcualted form 10 "weakest" gates. Now, there is a discontinuity at gate 65 in the FHW data so that either before or after that gate there is no power enhancement, and the respective 10 gates at the "normal" noise level are readily available. Furthermore, for some obscure reason the "streaks" are much more powerful at FHE and seem to last longer.

[sshepherd]

cve is more sensitive, in general, than fhw and sees higher power levels. Is the longer duration due to camping v no camping?

On Mon, Apr 10, 2017 at 10:57 PM pasha-ponomarenko notifications@github.com wrote:

This is not surprising because the noise is calcualted form 10 "weakest" gates. Now, there is a discontinuity at gate 65 in the FHW data so that either before or after that gate there is no power enhancement, and the respective 10 gates at the "normal" noise level are readily available. Furthermore, for some obscure reason the "streaks" are much more powerful at FHE and seem to last longer.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SuperDARNCanada/fitacf.3.0/issues/7#issuecomment-293135838, or mute the thread https://github.com/notifications/unsubscribe-auth/AC2DP7Sc2fPJ96KSBdYoLCnnP99aW33cks5ruuwrgaJpZM4Ml1Gm .

[pasha-ponomarenko]

I don't think so. Also, the apparently longer duration might just result from the higher power when even the "tail" of the perturbation is still well above the regular noise "floor".

[pasha-ponomarenko]

By the way, FITACF3 is still does not perform as well as expected with removing the "salt-and-pepper" variations as compared to the IDL prototype on my Windows laptop, so there is still something lost in the transition. We will have another look...

[egthomas]

I think the longer duration is simply because the CVE data is being plotted at 2-min resolution while the FHW data is being plotted at ~6-sec resolution.

I was just noticing the same thing about the salt-and-pepper noise while looking at some BKS data:

[pasha-ponomarenko]

Sorry, when I said "longer" I meant in terms of range, which is defined by time delay as well. Sometimes you see these things beyond gate 65, sometimes you see them before, less frequently -- on both sides. For CVE you generally see them on both sides.

With repsect to the "salt-and-pepper", I meant a bit different thing: the presence of high-velocity "spikes", both positive and negative, in low-power ground scatter (see top panel of my previous figure at the top of the page). What you have shown above is a bit different: just too much sporadic nosie-like data. This also looks like another issue to have a closer look at.

[pasha-ponomarenko]

By the way, I am a bit surprised to see a noticeable difference between FITACF2.4 and 2.5. To my understanding, there should not be that much, if any at all... Correction on 02 May 2017: actually, there should be a noticeable difference because the CRI threshold for rejecting bad lags incerases.

[egthomas]

The version of fitacf.2.4 that I'm using is from Simon's copy of RST 3.2 with a build date of early 2011. That may not be the final version, as the fitacf library in his copy of RST 3.3 from 2012 is also tagged as version 2.4

[pasha-ponomarenko]

I want to collect all previous versions of FITACF. Could you send me yours?

[egthomas]

Here are the versions of make_fit and fitacf we have for RST 2.11, 3.2, and 3.3:

fit_versions.tar.gz