BUG: Fan plot issues for files with multiple channel data

[carleyjmartin]

BUG

Issue plotting fan plots and rtp using data from PYK on 2015-03-08 noted by @Shirling-VT in last two releases.

Issue is rooted in how we grab a scan, in pyDARN we look for 1 in the scan flag to denote that a new scan has started, but on this day, the scan flag has a pattern (1,1,0,0,0,0....0,1,1,0,0...) where we see that it's only going to plot the first beam before thinking it's going to the next on the next beam. Resulting in only plotting one beam:

The scan mode is 153, and other 153 data seems to work fine, is this an issue with this specific radar/datafile/mode?

There is also an issue with beam 5 on this day for the same data, where it appears all data is uniform:

Again, is this an issue in the data or pyDARN specifically for this beam and day.

This time period appears to be fine for both issues on the VT website plotting tools that use DaViT, but we don't know if the data files are identical that are being used.

It would be good to test using our data files for DaViT (I believe @billetd is testing this) and I believe Pasha Ponomarenko (not in this repo?) is trying it out on his own software. Would be great to run it through RST too to see any differences - difference in plotting is that RST looks to plot 60 seconds of data and doesn't use the scan falg value like we do?

Priority

• [x] Minor - typo, incorrect naming, fails on specific uncommon situation - appears to be specific uncommon situation
• [ ] Major - misrepresentation of data, failure in a common situation
• [ ] Urgent - failure in usage, large misrepresentation

Information

python version: NA OS: Mac OS Catalina matplotlib version: 3.3.4 and 3.4.2

Potential Solution(s)

If it turns out this is a data issue, then it's out of our hands, but a proposed function to specify to plot only one beam and the user can build up a fan plot of any beams they like (also useful for camping beam ect).

I do not have any input on the beam 5 issue as it appears that might be a difference in data in the files?

If anyone has these issues on other data it would be great to find some more to test on!

Further discussion of this issue can be found in release PRs #181 and #166

[mts299]

Soooo looks like RST plots "scans" based on times and not the scan flag:

60 seconds

And yes you can plot 120 seconds, how who knows :p

So... I think there is no consistency on fan plots and being the DVWG we should maybe start ;)

@billetd this is you code so I will let you have the decision on how we solve this problem but I was thinking we could add the scan_time_length = 60s as an option for these things and maybe not allow anything to long like 120 s as I don't think that follow PI agreement :p

[egthomas]

Is this an issue of separating data on one channel from another for a Leicester-style Stereo radar where both channels are stored in the same file?

[Shirling-VT]

@egthomas @mts299 @carleyjmartin Evan made a really good point. I printed the content of this data file. As can be seen from the beam summary content below, the PYK radar was operating in two channels with different cpids (153 and -26999). And only beam 5 was collecting data in channel B. This may explain the issues with beam 5. So I think this is not a data file issue, it depends on how pydarn deals with the Leicester-style Stereo radar where both channels are stored in the same file as pointed out by Evan. ##################################################################################### time channel bmnum scan frang smsep rsep cp nrang mppul lagfr intt_sc intt_us sky_noise npts 2015-03-08 14:00:00.009999 1 0 1 180 300 45 153 70 7 1200 3 0 1194.9 24 2015-03-08 14:00:00.009999 2 5 1 180 300 45 -26999 70 7 1200 3 0 16.9 29 2015-03-08 14:00:03.119991 1 1 0 180 300 45 153 70 7 1200 3 0 1059.4 24 2015-03-08 14:00:03.119991 2 5 0 180 300 45 -26999 70 7 1200 3 0 16.6 34 2015-03-08 14:00:06.229984 1 2 0 180 300 45 153 70 7 1200 3 0 1252.0 23 2015-03-08 14:00:06.229984 2 5 0 180 300 45 -26999 70 7 1200 3 0 16.8 31 2015-03-08 14:00:09.349976 1 3 0 180 300 45 153 70 7 1200 3 0 1040.6 23 2015-03-08 14:00:09.349976 2 5 0 180 300 45 -26999 70 7 1200 3 0 16.9 31 2015-03-08 14:00:12.389973 1 4 0 180 300 45 153 70 7 1200 3 0 1130.4 23 2015-03-08 14:00:12.389973 2 5 0 180 300 45 -26999 70 7 1200 3 0 16.7 29 2015-03-08 14:00:15.499965 1 5 0 180 300 45 153 70 7 1200 3 0 1258.4 23 2015-03-08 14:00:15.499965 2 5 0 180 300 45 -26999 70 7 1200 3 0 16.6 32 2015-03-08 14:00:18.619957 1 6 0 180 300 45 153 70 7 1200 3 0 1305.2 23 2015-03-08 14:00:18.619957 2 5 0 180 300 45 -26999 70 7 1200 3 0 16.7 33 2015-03-08 14:00:21.729949 1 7 0 180 300 45 153 70 7 1200 3 0 1466.8 23 2015-03-08 14:00:21.729949 2 5 0 180 300 45 -26999 70 7 1200 3 0 16.7 32 2015-03-08 14:00:24.749948 1 8 0 180 300 45 153 70 7 1200 3 0 1288.4 22 2015-03-08 14:00:24.749948 2 5 0 180 300 45 -26999 70 7 1200 3 0 16.5 29 2015-03-08 14:00:27.869940 1 9 0 180 300 45 153 70 7 1200 3 0 1425.3 25 2015-03-08 14:00:27.869940 2 5 0 180 300 45 -26999 70 7 1200 3 0 17.1 32 2015-03-08 14:00:30.979932 1 10 0 180 300 45 153 70 7 1200 3 0 1914.9 23 2015-03-08 14:00:30.979932 2 5 0 180 300 45 -26999 70 7 1200 3 0 16.6 31 2015-03-08 14:00:34.099993 1 11 0 180 300 45 153 70 7 1200 3 0 2167.0 29 2015-03-08 14:00:34.099993 2 5 0 180 300 45 -26999 70 7 1200 3 0 16.7 33 2015-03-08 14:00:37.209985 1 12 0 180 300 45 153 70 7 1200 3 0 2269.3 26 2015-03-08 14:00:37.209985 2 5 0 180 300 45 -26999 70 7 1200 3 0 16.6 33 2015-03-08 14:00:40.319978 1 13 0 180 300 45 153 70 7 1200 3 0 1972.9 23 2015-03-08 14:00:40.319978 2 5 0 180 300 45 -26999 70 7 1200 3 0 16.1 32 2015-03-08 14:00:43.449969 1 14 0 180 300 45 153 70 7 1200 3 0 1660.0 21 2015-03-08 14:00:43.449969 2 5 0 180 300 45 -26999 70 7 1200 3 0 16.6 33 2015-03-08 14:00:46.559961 1 15 0 180 300 45 153 70 7 1200 3 0 1653.3 23

[mts299]

Thank you @egthomas and @Shirling-VT for pointing this out. We shall discuss at the DVWG meeting in August to figure how to solve this problem as there is multiple things to decide on how pyDARN will handle this kind of data.

[carleyjmartin]

For completeness, just updating here!

I've updated the name of the issue to reflect that it's a handling of multiple channels issue in fan plots.

The rtp beam 5 issue is sort of solved using the channel keyword for range time and time series plots which is already implemented, it's set to default to all channels which would need to be amended for these data. --> discussion topic for meeting would be is the 'all' default a good default?

Using:

import matplotlib.pyplot as plt 
import pydarn

file = "/Users/carley/Documents/data/20150308.1401.00.pyk.fitacf"
SDarn_read = pydarn.SuperDARNRead(file)
fitacf_data = SDarn_read.read_fitacf()

a = pydarn.RTP.plot_range_time(fitacf_data, beam_num=5, coord=pydarn.Coords.RANGE_GATE, channel=1)

I get:

[pasha-ponomarenko]

@mts299 , FYI, restricting scan time to 60 s will cause problems with all SD data recorded before mid 2007 as they were obtained with a 2-minute scan duration (7 s intergation time per beam). Indeed, a proper solution would be to separate channels as those are not supposed to be mixed/interleaved anyways.

And yes you can plot 120 seconds, how who knows :p

So... I think there is no consistency on fan plots and being the DVWG we should maybe start ;)

@billetd this is you code so I will let you have the decision on how we solve this problem but I was thinking we could add the scan_time_length = 60s as an option for these things and maybe not allow anything to long like 120 s as I don't think that follow PI agreement :p

[mts299]

@pasha-ponomarenko thank you

[billetd]

Agree with @pasha-ponomarenko that the solution probably shouldn't come in the form of using using the timescale of a scan. We can discuss this fully at the meeting, but I believe the best course will be implementing a channel keyword to plot_fan that has a default, similar to RTP plots like @carleyjmartin mentioned.

[mts299]

@billetd that should be easy to implement but I still like the idea of a time_length idea as extra flexibility. We don't know what the user will want and there may be modes that will use this. We don't need to make a cut-off time like I suggested before but just have the option to say plot 60 seconds or 10 or whatever.

We can discuss more in the meeting but I think the channel can be implemented ASAP

[mts299]

@carleyjmartin has assigned herself to solve this issue by implementing a channel option in plot_fan