CPID overlapping in RTI plot

[MuhammadVT]

As seen in the attached figure, the CPIDs are interlining with each other. As discussed with @asreimer in #185, CPID seems to be being plotted at every minute.

Here are the commands that reproduce the bug:

from davitpy.pydarn.plotting import rti import datetime as dt stime = dt.datetime(2010, 1, 15) etime = dt.datetime(2010, 1, 16) params = ['velocity'] coords = 'gate'; scales=[[-120,120]] rad = 'bks'

rad = 'sas'

bmnum = 7 channel = None file_type = 'fitex' colors = 'aj' gsct=True; low_gray=True file_name = None fig = rti.plot_rti(stime, rad, eTime=etime, channel=channel, params=params, colors=colors, gsct=gsct, low_gray=low_gray, bmnum=bmnum, coords=coords, fileName=file_name, fileType = file_type, scales=scales)

Here is the plot,

[asreimer]

The source of this problem (which seems rather obvious now) is that the cpid in the data file is switching back and forth from 153 to -26401. To see this, run the following code:

from davitpy import pydarn
import datetime as dt
stime = dt.datetime(2010, 1, 15)
etime = dt.datetime(2010, 1, 16)
rad = 'bks'
bmnum = 7
myPtr = pydarn.sdio.radDataOpen(stime,rad,eTime=etime,bmnum=bmnum)
for i in range(10):
    print myBeam.cp, myBeam.bmnum, myBeam.time

So, what do we do about this? We want to display the cpid, and it alternates like that, how do we get around this? I can't really think of anything that wouldn't simply produce the same "black bar" output that we are seeing now.

The best thing to do here is simply specify the cpid while plotting to "filter" based on cpid. For example:

from davitpy.pydarn.plotting import rti
import datetime as dt
stime = dt.datetime(2010, 1, 15)
etime = dt.datetime(2010, 1, 16)
params = ['velocity']
coords = 'gate'; scales=[[-120,120]]
rad = 'bks'
bmnum = 7
channel = None
file_type = 'fitex'
colors = 'aj'
gsct=True; low_gray=True
file_name = None
cpid = 153
myPtr = pydarn.sdio.radDataOpen(stime,rad,eTime=etime,bmnum=bmnum,cp=cpid)
fig = rti.plot_rti(stime, rad, eTime=etime, channel=channel, params=params, colors=colors, gsct=gsct, low_gray=low_gray, bmnum=bmnum, coords=coords, fileName=file_name, fileType = file_type, scales=scales,myFile=myPtr)

Here's the plot I get doing this:

[MuhammadVT]

Hey @asreimer ,

Though specifying the cpid does fix the problem, it may not be an easy option to take for an outside user who is not quite familiar with cpid. The attached is the rti plot obtained from Superdarn website. As you can see, it simply listed the two alternating cipds one time during the fast switching period. What do you think if we do the similar thing to this pull request?

[asreimer]

That looks like it could work. I would be in favour of doing something like that. The only thing I would be worried about with it is how we detect that things are oscillating quickly back and forth between the different CPIDs. I'm in favour of a general method that "just works" for any CPID situation.

Really, this is only a problem for this plot because one is plotting an entire day work of data. If you only plot 1 hour, then you can differentiate between CPIDS. I mean, what happens if you try to plot an entire 7 days worth of data and there are tons of CPIDs switching because of storms and special experiments running? With the listing solution that you suggested, there might be 5 or 6 CPIDs listed and then the plot is full of wasted white space trying to list CPIDs. The point it there will always be some issue with trying to differentiate between CPIDs depending on the time scale between switching CPIDs and the time scale of the rti plot. At this point, the noise, nave, and tx freq are also going to be tough to distinguish between.

Trying to detect when CPIDs are switching quickly with really slow down the rti plotting... Trying to replace labels with colours or symbols will result in the same problem as what we currently have where things all overlap.

So TL;DR: I think you are on to something there @MuhammadVT, but we have to be careful how we implement it for the reasons I just talked about. A good start would be to assign a letter or symbol for each CPID that is plotted and then have a list of the CPIDs and their corresponding symbols off to the side of the plot or something. That way, even if the CPID plot is a mess because the symbols are overlapping, at least the user will be able to see that there is more than one CPID being plotted and that is the reason for it.

[aburrell]

What if, in cases where there are multiple CPIDs we add a text warning to the plot (either in the CPID area, since that’s empty now or at the top)? Then people can see what modes are running and make a new plot using the data they want, if they can’t distinguish the modes in whatever plot they made. I really don’t think it’s a good idea to have the program self select. As Ashton says, it will slow things down and may also cause problems if people want to see all the modes for a long period of time.

On 3 Dec 2015, at 22:22, Ashton Reimer notifications@github.com wrote:

That looks like it could work. I would be in favour of doing something like that. The only thing I would be worried about with it is how we detect that things are oscillating quickly back and forth between the different CPIDs. I'm in favour of a general method that "just works" for any CPID situation.

Really, this is only a problem for this plot because one is plotting an entire day work of data. If you only plot 1 hour, then you can differentiate between CPIDS. I mean, what happens if you try to plot an entire 7 days worth of data and there are tons of CPIDs switching because of storms and special experiments running? With the listing solution that you suggested, there might be 5 or 6 CPIDs listed and then the plot is full of wasted white space trying to list CPIDs. The point it there will always be some issue with trying to differentiate between CPIDs depending on the time scale between switching CPIDs and the time scale of the rti plot. At this point, the noise, nave, and tx freq are also going to be tough to distinguish between.

Trying to detect when CPIDs are switching quickly with really slow down the rti plotting... Trying to replace labels with colours or symbols will result in the same problem as what we currently have where things all overlap.

So TL;DR: I think you are on to something there @MuhammadVT https://github.com/MuhammadVT, but we have to be careful how we implement it for the reasons I just talked about. A good start would be to assign a letter or symbol for each CPID that is plotted and then have a list of the CPIDs and their corresponding symbols off to the side of the plot or something. That way, even if the CPID plot is a mess because the symbols are overlapping, at least the user will be able to see that there is more than one CPID being plotted and that is the reason for it.

— Reply to this email directly or view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/186#issuecomment-161804617.

[MuhammadVT]

Hey @asreimer,

Are you still working on this issue? If not, then we may keep this issue open and merge #185 for the time being.

[asreimer]

Hey @MuhammadVT,

I'm not working on this issue in #185. We should fix this issue though!

[ksterne]

Hey all,

I've got something working in the #266 branch though I haven't pushed it up and it's just on my local computer. The result test case is:

the error message that pops up is:

ERROR:root:CPID is changing too frequently to be legibly printed. Please consider using radDataOpen cp param. CPIDs found: [153, -26401]

For now, I have this pop up if the cpid changes more than 20 times which seems like it would be too much to be readable on the panel. Any thoughts? If you'd like to see it, I can push it up to the rti enhancement branch.

[asreimer]

Ya that's a nice simple solution.

Perhaps the threshold for the "error" text can be set by a keyword passed to plot_rti but default to 20?

Also, maybe we can change the error text to something like "Threshold exceeded. Changing too frequently to display" or something like that but short and concise? If I think of something better I'll post back here.

[ksterne]

Ok, sounds like we have a start at least. I've added this to the #266. We can leave this issue open using #266 is merged in which should fix/address it.

[ksterne]

This can be closed as #266 was merged in a while ago.