search

vtsuperdarn / VTRST3.5

Radar Software Toolkit enhanced
0 stars 6 forks source link

Tigfitex #5

Closed ksterne closed 10 years ago

ksterne commented 10 years ago

Found tig data from 20120308 that can have a large negative value for the lag table. So, here doing a check to see if the lag table is negative (it should never be negative right?). If it is negative, then set it to zero (what I think it was really supposed to be) and spit out a warning about it. Maybe the warning message is a little overboard.

ksterne commented 10 years ago

Also found that if the lag is 100 or greater then the process segment faults since it's trying to access a value outside of the lag_flg array. For now, putting a continue in for this. Found this happens on 20120810.0405tig rawacf file.

ksterne commented 10 years ago

Moving this to another branch, continuing to work on it.

pasha-ponomarenko commented 10 years ago

Hi Kevin! I am not very familiar with fitex, so please excuse me if I ask silly questions. If understand it right, FITEX is just a different way of fitting the phase, so that it uses the same raw data as FITACF. In this case the lag table contains pairs of pulses used for generating specific ACF lags. Each pulse is described by its emission time with respect to the beginning of the pulse sequence expressed in a number of elementary lags, prm. mpinc, which should always be positive. So, my questions are: (1) In your opinion, what sort of error could produce these negative values? (2) Why do you think the negative pulse numbers should be replaced by just zeroes but not by their absolute values? (3) Should we also fix this problem in FITACF which uses the same raw data?

Cheers, Pasha

On 01/09/2014 1:43 PM, Kevin Sterne wrote:

Found tig data from 20120308 that can have a large negative value for the lag table. So, here doing a check to see if the lag table is negative (it should never be negative right?). If it is negative, then set it to zero (what I think it was really supposed to be) and spit out a warning about it. Maybe the warning message is a little overboard.

    You can merge this Pull Request by running

git pull https://github.com/vtsuperdarn/VTRST3.5 tigfitex

Or view, comment on, or merge it at:

https://github.com/vtsuperdarn/VTRST3.5/pull/5

    Commit Summary

— Reply to this email directly or view it on GitHub https://github.com/vtsuperdarn/VTRST3.5/pull/5.

Dr Pavlo (Pasha) Ponomarenko

Professional Research Associate Institute of Space and Atmospheric Studies University of Saskatchewan

Mailing address: Physics building 116 Science Place, Saskatoon, SK, S7N 5E2 CANADA Phone +1 306 966-6458 Facsimile +1 306 966-6440 Email pasha.ponomarenko@usask.ca WWW www.usask.ca/~pasha.ponomarenko

 The opposite of skeptic is gullible.

I think it’s much more interesting to live not knowing, than to have answers which might be wrong.

Richard Feynman

The object of life is not to be on the side of the majority, but to escape finding oneself in the ranks of the insane.

Marcus Aurelius Augustus

pasha-ponomarenko commented 10 years ago

HI Kevin First, a bit of clarification: the prm.lag parameter represents a two-dimensional array containing pairs of pulses which are used for generating ACF lags (lags are recorded in an increasing order). The difference between its values for the first and the second pulses will be the lag value measured in the elementary lags, prm.mpinc.

I had a closer look at this parameter for TIGER data at 04-06UT on 10 August 2012, which was mentioned in your request. Indeed, between 4 and 5UT the lag table sporadically shows erroneous values for the first two lags, lag 0 and lag 1 which can be very large in magnitude and even have a negative sign. I am not sure why this has happened, bu the fix should be a bit different form yours, i.e. filling the table with zeroes instead of the "wrong" values. Indeed, for the prm.lag[0,0] and prm.lag[0,1] the values should be 0 and 0 because for the lag 0 for the given pulse sequence (katscan) we use the very first pulse. However, for the first lag we use pulses number 43 and 42 so that replacing these values by zeroes would distort the respective ACFs.

I did not look at the second day, 8 March 2012 becasue the respective fitacf file is absent from our archive, but it looks like it has the same problem. If this is so, then I believe that the best fix would be
to fix the data files rather than to change the analysis software, although I wonder how widespread the problem really is.

Cheers, Pasha

On 02/09/2014 1:09 PM, Kevin Sterne wrote:

Moving this to another branch, continuing to work on it.

— Reply to this email directly or view it on GitHub https://github.com/vtsuperdarn/VTRST3.5/pull/5#issuecomment-54202050.

Dr Pavlo (Pasha) Ponomarenko

Professional Research Associate Institute of Space and Atmospheric Studies University of Saskatchewan

Mailing address: Physics building 116 Science Place, Saskatoon, SK, S7N 5E2 CANADA Phone +1 306 966-6458 Facsimile +1 306 966-6440 Emailpasha.ponomarenko@usask.ca WWWwww.usask.ca/~pasha.ponomarenko

 The opposite of skeptic is gullible.

I think it’s much more interesting to live not knowing, than to have answers which might be wrong.

Richard Feynman

The object of life is not to be on the side of the majority, but to escape finding oneself in the ranks of the insane.

Marcus Aurelius Augustus

ksterne commented 10 years ago

Pasha,

Sorry for not responding a little quicker. Many things on my plate and more so wanting to go through all of the rawacf years and see what else may need to be fixed. To answer your previous questions:

1) I'm not at all sure. I'm guessing thats some kind of error in memory addressing. I haven't talked with anyone at La Trobe about the error and what may be causing it. 2) I put in zeros originally here as a way to test them. I agree with your second note that doing this will degrade the acfs, but I'm not quite sure how we could replace it aside from hardcoding the lag table in. From what I can remember, this large negative value can happen on just more than the zeroth or first lags. I don't think any kind of hardcoding is a good idea though. 3) I haven't looked at how FITACF deals with this data. I do know that it doesn't crash in the same manner as fitex2. That would be an interesting thing to explore.

As you'll see here, I ended up closing this pull request after I found a few more issues that aren't specific to just the Tiger data. I've been making a few bits of work in another branch and will put in a pull request when I think that it's good. I agree here that maybe this is an issue where the rawacf files should be corrected, but first I'd have to see how often it happens. There are a few things that can be done without changing the lag table that would keep the fitex2 routine from crashing.

Cheers, -Kevin