fitexFilter routine hardwired to run a single radar and time

[aburrell]

The fitFilter routine in fitexfilter is hardwired to run Blackstone data on a single day. Here is the fix:

` def fitFilter(stime, rad, outfile, thresh=0.4, infile=None, etime=None, channel=None, bmnum=None, cpid=None, src=None, nocache=False, local_dirfmt=None, local_fnamefmt=None, local_dict=None, remote_dirfmt=None, remote_fnamefmt=None, remote_dict=None, remote_site=None, username=None, password=None, port=None, tmpdir=None): """This function applies a boxcar filter to a fitacf file

Parameters
------------
stime : (dt.datetime object)
    Starting time of input file
rad : (str)
    3 character radar id code
outfile : (str)
    The name of the output file
thresh : (float)
    The filter threshold for turning on a R-B cell.  (default=0.4)
infile : (str)
    The name of the input fitacf-format file
etime : (datetime/NoneType)
    The last time that you want to input data for.  If this is set to None,
    it will be set to 1 day after sTime.  (default=None)
channel : (str/NoneType)
    The 1-letter code for what channel you want data from, eg 'a','b',...
    if this is set to None, data from ALL channels will be read.
    (default=None)
bmnum : (int/NoneType)
    The beam number which you want data for.  If this is set to None, data
    from all beams will be read. (default=None)
cpid : (int)
    The control program which you want data for.  If this is set to None,
    data from all cp's will be read.  (default=None)
src : (str/NoneType)
    The source of the data.  Valid inputs are 'local' 'sftp'.  If this is
    set to None, it will try all possibilites sequentially.  (default=None)
infile : (str/NoneType)
    The name of a specific file which you want to open.  (default=None)
nocache : (boolean)
    Flag to indicate that you do not want to check first for cached files.
    (default=False)
remote_site : (str/NoneType)
    The remote data server's address.  If None, the rcParam value DB will be
    used. (default=None)
port : (str/NoneType)
    The port number to use for remote_site.  If None, the rcParam value
    DB_PORT will be used. (default=None)
username : (str/NoneType)
    Username for remote_site.  If None, the rcParam value DBREADUSER will
    be used.
password : (str/bool/NoneType)
    Password for remote_site. If password is set to True, the user is
    prompted for the remote_site password.  If set to None, the rcParam
    value DBREADPASS will be used (default=None)
remote_dirfmt : (str/NoneType)
    The remote_site directory structure. Can include keywords to be
    replaced by dictionary keys in remote_dict.  If None, the rcParam value
    DAVIT_REMOTE_DIRFORMAT will be used. (default=None)
    Ex) remote_dirfmt='/{year}/{month}'
remote_fnamefmt : (str/list/NoneType)
    The remote_site file naming format. Can include keywords to be replaced
    by dictionary keys in remote_dict. If None, the rcParam value
    DAVIT_REMOTE_FNAMEFMT will be used.  (default=None)
    Ex) remote_fnamefmt=['{date}.{radar}.{ftype}',
                         '{date}.{channel}.{radar}.{ftype}']
local_dirfmt : (str/None)
    The local directory structure. Can include keywords to be replaced by
    dictionary keys in remote_dict. If None, the rcParam value
    DAVIT_LOCAL_DIRFORMAT will be used. (default=None)
    Ex) local_dirfmt='/{year}/{month}' 
local_fnamefmt : (str/list/NoneType)
    The local file naming format. Can include keywords to be replaced by
    dictionary keys in remote_dict. If None, the rcParam value
    DAVIT_LOCAL_FNAMEFMT will be used. (default=None)
    Ex) local_fnamefmt=['{date}.{radar}.{ftype}',
                        '{date}.{channel}.{radar}.{ftype}']
tmpdir : (str/NoneType)
    The directory in which to store temporary files. If None, the rcParam
    value DAVIT_TMPDIR will be used. (default=None)

Returns
---------
Void

Note
-----
This is VERY slow

written by AJ, 20130402
"""
from davitpy import pydarn
inp = pydarn.sdio.radDataOpen(stime, rad, eTime=etime, channel=channel,
                              bmnum=bmnum, cp=cpid, fileType='fitacf',
                              filtered=False, src=src, fileName=infile,
                              noCache=nocache, local_dirfmt=local_dirfmt,
                              local_fnamefmt=local_fnamefmt,
                              local_dict=local_dict,
                              remote_dirfmt=remote_dirfmt,
                              remote_fnamefmt=remote_fnamefmt,
                              remote_dict=remote_dict,
                              remote_site=remote_site, username=username,
                              password=password, port=port, tmpdir=tmpdir)

if inp._radDataPtr__filename is None:
    logging.warning("no data to filter for {:s} at {:}".format(rad, stime))
    return

outp = open(outfile, 'w')

scans = [None, None, None]

scans[1] = pydarn.sdio.radDataReadScan(inp)
scans[2] = pydarn.sdio.radDataReadScan(inp)
sc = scans[2]

while sc is not None:
    tsc = doFilter(scans, thresh=thresh)

    for b in tsc:
        logging.info("processing: {:}".format(b))
        pydarn.dmapio.writeFitRec(b, utils.datetimeToEpoch(b.time), outp)

    sc = pydarn.sdio.radDataReadScan(inp)
    scans[0] = scans[1]
    scans[1] = scans[2]
    scans[2] = sc

tsc = doFilter(scans, thresh=thresh)
logging.info("current at time: {:}".format(tsc.time))

for b in tsc:
    pydarn.dmapio.writeFitRec(b, utils.datetimeToEpoch(b.time), outp)

outp.close()
return

`

[asreimer]

Does anyone actually use this? Do we need it?

[ksterne]

A few people here in our lab use it and have asked about it. @aburrell, would you like to do the honors of getting this pull request into the new develop?

[asreimer]

Sounds good. Just making sure it's actually worth keeping :)

[aburrell]

Sure, I'll put this on my schedule for next week.

[ksterne]

Hey @aburrell, any movement on this?

[aburrell]

Sorry, I’m not going to be able to get to any davitpy things for about a month. Feel free to take over.

On 25 Mar 2016, at 17:26, Kevin Sterne notifications@github.com wrote:

Hey @aburrell https://github.com/aburrell, any movement on this?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/213#issuecomment-201377448

[ksterne]

Started taking a look at this and have a branch up and going with your fixes. Not only did that need a fixing, but doFilter has an awesome line right after the import that clears out the required variable scans ....doh!

[ksterne]

Ok, this is truely bizarre. There is a loop at the beginning of doFilter in the lines linked here. Every time the loop runs it adds another item onto the scans list, so there is always another s. I think I've replaced it with the lines above it.

Working out the looping issues...this function still doesn't seem to work as it's trying to call pydarn.dmapio.writeFitRec() which doesn't seem to exist in the dmapio dictionary... I believe this is where the function is trying to write a dmap file, but I don't think we have that functionality.

Calling @ajribeiro, calling @ajribeiro, can you help some poor souls out here? It seems as though the fitexFilter function isn't working too well.

Otherwise, I've added a warning that this function has fallen into disrepair. I'm not sure we want to remove this code altogether as it represents some good work AJ did. But in its current state, it's not functional. Any thoughts of how to move forward? Anyone is welcome to take a look at the bugfix-fitexFilter branch.

[ksterne]

I'm guessing there won't be any response from @ajribeiro....

I'm still wondering on how to move forward from here. Without a dmap write function, this won't do very good. @asreimer, @kkotyk, does the new python dmap include a write function?

For now, I've added try/except lines here to keep the file from crashing on its own.

[asreimer]

The new dmap library will include a write function.

Honestly, does anyone use this? Is there any documentation anywhere about what it is supposed to do even?

If this code has a purpose and people are using it, then let's keep it for sure, but if not, I'm ok with us just removing it.

[ksterne]

@asreimer, again, I think it's unwise of removing this. I get the need to slim things down and prevent bloatware. However, since this was a decent amount of work, I think it's unwise to completely lose it. As well, it's the only insight to a binary file that's floating around the RST/C world. Yes, it's true that there is no one that can speak to this, but I believe it can be useful and should not be removed.

Happy to hear of a write function with the new python dmap library.

[salsrag]

This comment applies to the many E-mails that I am receiving. I am concerned that this development >so running open loop. Could someone run the various C codes that are being translated and produce printouts or plots of the outputs at the various stages. The need to be E-mails to the more senior people within SuperDARN along with a request to identify the objective and importance of routine. Make sure that Mike, Bill, I are on the distribution.  Other should be added at your discretion. Ask me if have any questions. 

Sent via the Samsung Galaxy S® 6, an AT&T 4G LTE smartphone-------- Original message --------From: Kevin Sterne notifications@github.com Date: 4/21/2016 9:09 PM (GMT+01:00) To: vtsuperdarn/davitpy davitpy@noreply.github.com Subject: Re: [vtsuperdarn/davitpy] fitexFilter routine hardwired to run a single radar and time (#213) I'm guessing there won't be any response from @ajribeiro....

I'm still wondering on how to move forward from here. Without a dmap write function, this won't do very good. @asreimer, @kkotyk, does the new python dmap include a write function?

For now, I've added try/except lines here to keep the file from crashing on its own.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub

[aburrell]

@asreimer @ksterne , did we ever get a write function? Is it sensible to try and pick this up again?