Channel keyword in radDataPtr

[kherli]

Hi, I was just wondering about the status and plans for the channel keyword within radDataPtr. I can see that at the moment the description is: channel : (str/NoneType) The 1-letter code to specify the UAF channel (not stereo), e.g. 'a','b',... If 'all', ALL channels were obtained. (default=None, meaning don't check for UAF named data files)

I understand that this is applying to radars with only one channel. I am working with the stereo Longyearbyen radar and running each of the two channel independently, so would like to work with each channel separately. I can manually filter the channels to get either A or B myself, but can't use the radDataPtr to achieve this. Because of this, I can't use the davitpy fan plotting and RTI plotting as they currently are to plot only one channel. I can work around the problem using the RTI by specifying a CPID but I can't see how I can use the fan plotting without modifying it's contents.

Are there any plans to allow filtering of the stereo radar channels with radDataPtr?

Thanks!

[aburrell]

It's not on my list at the moment. Any chance you'd be willing to give it a shot @kherli ? I'd be happy to help you, but can't take it on for any time in the foreseeable future.

[kherli]

I can try and do it :) but I'd need to confirm a few things and I'm a bit confused about what has been previously tried. As it is, it looks like calling the channel keyword doesn't work at all for both mono and stereo radars and results in no data being found by the pointer.

I can see lots of places where the keyword used to be checked which are commented out, eg:

REMOVED BY ASR on 11 SEP 2014

    # the channel attribute in fitted files (fitacf, lmfit, fitex) specifies
    # if the data came from a STEREO radar, so we shouldn't clobber the 
    # value from the dmap file.
    #    elif(attr == 'channel'):
    #      if(aDict.has_key('channel')): 
    #        if(isinstance(aDict.has_key('channel'), int)):
    #          if(aDict['channel'] < 2): self.channel = 'a'
    #          else: self.channel = alpha[aDict['channel']-1]
    #        else: self.channel = aDict['channel']
    #      else: self.channel = 'a'
    #      continue

So I thought it would be best to check here first to see if there was something planned. I am familiar with the LYR radar's stereo operations (1=channel A, 2=channel B) but not so much with what 1 and 2 mean for a mono-radar. From looking at data from SAS, it looks to me as if the channel alternates between 1 and 2 every time a new scan is started, but I don't know if that is the same for all mono-radars. I also can't find anywhere on the VT website a list of which radars are in mono or stereo operation to confirm anything. Do you know of any place where this info is?

[aburrell]

These are good questions that I can't answer. @asreimer , @Shirling-VT , @ksterne would you know?

[egthomas]

The Canadian radars alternate between channels 1 and 2 according to the scan frequency being used when they operate the "twofsound" control program. This was practice began sometime in November 2016 so we could more easily treat data collected on either the higher or lower frequency as an independent data stream (but with 2-min cadence). So SAS, PGR, CLY, INV, RKN are not actually stereo/multi-channel radars; the channel=1/2 values stored in the parameter block are due to the control program.

[kherli]

Ah okay I see, thanks for explaining! I suppose the reason the code for current channel keyword has been commented out is that it could cause confusion if the user doesn't know the difference between the two ways the frequency is switched between stereo and single channel radars. (Please confirm @asreimer ? I guess from the initials in the comments that's you :) )

I am not sure what other people are usually using the channel keyword for and what people want from it. I guess we could add a list of radars in stereo operation and only allow the channel keyword to work on those radars and note this in the explanation of the keyword that it only applies to stereo radars and that in all other cases all data is returned.

That would be what I was looking for, but I don't know if others would find it useful to be able to also filter the single-channel radars like that to extract the data ran at each frequency. In which case it would make sense to bring back the commented code and maybe add to the explanation what 1 and 2 mean in the case of stereo and non-stereo operations.

If neither one of these options seems good then the keyword should be removed I guess since it isn't doing anything. What do you guys think?

[Shirling-VT]

Usually, I use the following code to deal with the radar channel issue in davitpy, when I just want data from channel A from all radars.

if rad in ['ksr','kod','ade','adw','mcm','sps']: channel = 'a' else: channel = None

myPtr = pydarn.sdio.radDataOpen(sTime,rad,eTime=eTime,channel=channel,cp=cp,fileType=fileType,filtered=filtered, src=src)

Hope this helps with your question of the channel keyword usage.

On Mon, Oct 2, 2017 at 6:11 AM, kherli notifications@github.com wrote:

Ah okay I see, thanks for explaining! I suppose the reason the code for current channel keyword has been commented out is that it could cause confusion if the user doesn't know the difference between the two ways the frequency is switched between stereo and single channel radars. (Please confirm @asreimer https://github.com/asreimer ? I guess from the initials in the comments that's you :) )

I am not sure what other people are usually using the channel keyword for and what people want from it. I guess we could add a list of radars in stereo operation and only allow the channel keyword to work on those radars and note this in the explanation of the keyword that it only applies to stereo radars and that in all other cases all data is returned.

That would be what I was looking for, but I don't know if others would find it useful to be able to also filter the single-channel radars like that to extract the data ran at each frequency. In which case it would make sense to bring back the commented code and maybe add to the explanation what 1 and 2 mean in the case of stereo and non-stereo operations.

If neither one of these options seems good then the keyword should be removed I guess since it isn't doing anything. What do you guys think?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/323#issuecomment-333493781, or mute the thread https://github.com/notifications/unsubscribe-auth/AIEZKwISzV1e3Y8pSqnrxv-bCM5r2hxMks5soLbIgaJpZM4Pohxs .

-- Best wishes, Xueling

[kherli]

Thanks for your answer @Shirling-VT! Do you know why my following code returns both channels when I specify channel=None but no data when I specify channel = 'a' or channel = 'b' ? This result happens for all radars and dates for me.

myPtr = pydarn.sdio.radDataOpen(sTime, rad, eTime=eTime, fileType='fitacf', src='local', local_dirfmt='/home/katie/work/Superdarn_data/data/{year}/{ftype}/{radar}/')
myData=radDataReadAll(myPtr)

When I look at the data returned when I specify channel = None, I can see both channel 1 and 2 data in there. I am currently reading in all data and then filtering it by channel myself afterwards by reading each record separately. But now I'm not sure why my above code doesn't work if then channel keyword is working for you and successfully filtering channel A...

[Shirling-VT]

Taking a careful look at my code which was implemented a year ago for searching THEMIS mode data, the way I took care of channel only considering the UAF radars with channel names specified in the filename. For stereo operation, data from multiple channels are stored in the same file. Since I am focused on the THEMIS mode data, I used the CPID to filter out the data I want after reading all the records from all channels, the same way you did for your RTI plot. Sorry, I cannot figure out a better way of solving this issue at this moment.

On Mon, Oct 2, 2017 at 11:43 AM, kherli notifications@github.com wrote:

Thanks for your answer @Shirling-VT https://github.com/shirling-vt! Do you know why my following code returns both channels when I specify channel=None but no data when I specify channel = 'a' or channel = 'b' ? This result happens for all radars and dates for me.

myPtr = pydarn.sdio.radDataOpen(sTime, rad, eTime=eTime, fileType='fitacf', src='local', local_dirfmt='/home/katie/work/Superdarn_data/data/{ year}/{ftype}/{radar}/') myData=radDataReadAll(myPtr)

When I look at the data returned when I specify channel = None, I can see both channel 1 and 2 data in there. I am currently reading in all data and then filtering it by channel myself afterwards by reading each record separately. But now I'm not sure why my above code doesn't work if then channel keyword is working for you and successfully filtering channel A...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/323#issuecomment-333574611, or mute the thread https://github.com/notifications/unsubscribe-auth/AIEZK-JV7Z3B6VM1EMWe-u0sgtvQ_sCmks5soQStgaJpZM4Pohxs .

-- Best wishes, Xueling

[aburrell]

To summarize (and make sure I'm following this correctly), it seems that channel as input only works when it is specified in the filename and not in the file itself. So what we need is a two step process, where the code first looks to separate data by filename and then looks to restrict the data using the file information itself (as is done for cpid and beam number). The logic will be a bit tricky (since we should avoid setting certain processing modes for certain named radars), but doable. Does that sound about right, @kherli ?

[kherli]

Thanks @Shirling-VT! I actually didn't know what UAF stood for so didn't fully understand what the keyword was doing, but after reading your code I now realize that it works for all University of Alaska Fairbanks radars as it says in the description. Do you know what channel 1 and 2 means for the UAF radars? If you say they're not stereo, are they operating like the Canadian radars Evan described earlier on this thread?

@aburrell I'm also still trying to fully understand how the the different conventions used (channel number, filename) differ between radars but so far what you've summarized is also my understanding too :)

[Shirling-VT]

@kherli https://github.com/kherli , I have no idea what channel 1 and 2 means here, usually I use channel 'a','b','c','d' in davitpy and take it granted that they correspond to number 0,1,2,3. But as Evan pointed out, they have different meanings in RST and the dmap files.

To the best of my understanding after talking with Kevin, there are three categories of radars operating in multi channels:

1. UAF radars (ADW / ADE / KOD / MCM / SPS) and KSR sometimes, as I mentioned before, when they use two channels to do the scan, the channel name will be specified in the filename, such as 20160108.0402.00.ksr.a.fitacf.bz2. The channel keyword set to a, b , c or d in davitpy reading code works for these radars when operating in specific channel.

2. Stereo operation radars (could be HAN / PYK / STO / HKW / LYR / BKS ...): operating in two channel simultaneously but stored in one zipped file which makes the channel keyword not working, one way to separate data from different channels in davitpy is filter data out using CPID.

3. Canadian radars (SAS, PGR, CLY, INV, RKN ): as Evan pointed out, they alternate between channels 1 and 2 according to the scan frequency being used when they operate the "twofsound". I don't think the channel keyword would work for them as well.

Not all the radars I specified in three categories always operate using multi channels, these make things even more complicated. Please comment on my above understanding.

On Mon, Oct 2, 2017 at 3:13 PM, kherli notifications@github.com wrote:

Thanks @Shirling-VT https://github.com/shirling-vt! I actually didn't know what UAF stood for so didn't fully understand what the keyword was doing, but after reading your code I now realize that it works for all University of Alaska Fairbanks radars as it says in the description. Do you know what channel 1 and 2 means for the UAF radars? If you say they're not stereo, are they operating like the Canadian radars Evan described earlier on this thread?

@aburrell https://github.com/aburrell I'm also still trying to fully understand how the the different conventions used (channel number, filename) differ between radars but so far what you've summarized is also my understanding too :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/323#issuecomment-333635225, or mute the thread https://github.com/notifications/unsubscribe-auth/AIEZK5dQlqkkbr3u8MuH5S8uUEjXtGyzks5soTW8gaJpZM4Pohxs .

-- Best wishes, Xueling

[aburrell]

Nice summary @Shirling-VT . It looks like we've all very organized and consistent within the SD community! So there are three ways to separate the data, and depending on what radar you're using, one or more of them will be appropriate:

1) channel keyword in the filename (UAF radars, sometimes KSR). It looks like this behaviour is currently supported 2) CPID. This behaviour is also supported, but there could be more documentation explaining that Stereo operations have different CPIDs for the different channels. It would also be nice to have a list of the different CPIDs with descriptions. 3) Frequency. We don't currently support data filtering by frequency. This would be nice, but tricky. If all radars used specific frequency bands, it would be fine, but they don't. I've played around with this (check out pydarn.radar.tdiff.rad_freqbands), and it would be nice to have as an option. Again, it would be good to have more documentation on this.

[kherli]

Thanks @Shirling-VT! I can verify the (2.) Stereo point for LYR at least since that's what I'm familiar with. When I originally asked the question I didn't realise how complicated it would be to find a channel keyword which will work for all radars at all times. Thanks for enlightening me :)

@aburrell I've just worked around my fanplot problem by changing the contents of the file and filtering for stereo radar channels. I'm not sure I'd know how to make a channel keyword which would work for everyone, it might be a better idea to allow an optional myPtr keyword to the fanplot which would allow you to pass it the data you want filtered by channel.

[asreimer]

Sorry, I'm trying to catch up on things. @kherli yes, I was the one who removed the channel code. This is because of the ambiguity in what channel means. As already mentioned here, channel is several different things depending on the radar you use and this is largely because the SuperDARN community has played fast and loose with how the data is stored.

So, I removed that code because for stereo radars it was clobbering the channel number. I encourage anyone looking to modify the code to first read issue #96 for some context (also see #93). That is why I commented out the channel code. I figured we might want to reuse or build upon it at some point though, so I didn't delete it, only commented it out.

But wait! There's more! This issue is actually more complicated for davitpy than just simply the issues with the SuperDARN community playing fast and loose with what channel number means. In davitpy, the radDataPtr doesn't just read files, it is also used to automagically fetch and download files. So of course, it also only grabs what you want it to grab (or at least tries to). The "channel" parameter was intended to be used to help filter which files you grab too. This means that "channel" in davitpy after issue #96 was designed to only help fetch UAF and non-UAF naming convention files.

In practice, I find this is actually not how most people analyze data. So, the whole data fetching and reading for davitpy needs to be rewritten. Specifically, fetching and reading need to be two different things, meaning file fetching needs to be pulled out of the radDataPtr code.

I think we need to have a developers meeting to hash this issue out better. It would be nice if the PIs would agree on a consistent file format, but we can definitely design file reading to be mostly agnostic to file contents. We can also change the intended usage of davitpy so that people understand they need to fetch the data they want to use independently of reading and processing it (which seems to be the way everyone I know uses davitpy anyway).

[kherli]

Thanks for the details @asreimer! I can see why the commented out code is still around now and from everyone's helpful details I understand more about what the keyword actually does and what the issues are with the file naming conventions. I have my workaround for now, but what you say about separating the fetching/reading/processing makes sense, I am fetching the data myself outside of davitpy anyways :)