Enhancement rst 4 1 data types

[ksterne]

This pull request is to update davitpy with the new file types available with RST 4.1. These are fitacf3 (though optional for now), grid2, and map2. I've also updated the sdDataTypes "unit" test section which must have been out of date when grid/map files were reprocessed. In addition to fixing the current tests there, I added ones that will pull the new grid2, map2 files as well as added a section to the radDataTypes "unit" test section to include pulling a fitacf3 file. The focus here was mostly to enable the new files types in the sdio/ directory, but I also added the new files types in some of the documentation in the plotting/ directory.

Otherwise, please use these codes to test things here:

from matplotlib import pyplot as plt
from pylab import *
import datetime as dt
from davitpy import pydarn
#from davitpy.utils import geoPack

t1=dt.datetime(2018,1,17)
t2=dt.datetime(2016,8,22,00,00)

rad='fhe'
bmnum=13
rnglimit=[0, 3500]

fig=pydarn.plotting.rti.plot_rti(t1,rad,bmnum=bmnum,params=['velocity'],coords='rng',yrng=rnglimit,gsct=True, fileType='fitacf3')

#fig.show()
savefig('rti_output.png')

results in:

import datetime
import matplotlib.pyplot as plt
from pylab import *
import davitpy.pydarn.plotting.plotMapGrd
from davitpy.utils import *

fig = plt.figure()
ax = fig.add_subplot(111)

sdate = datetime.datetime(2017,4,3,4,0)
mObj = plotUtils.mapObj(boundinglat=50., gridLabels=True, coords='mag')

mapDatObj = davitpy.pydarn.plotting.plotMapGrd.MapConv(sdate, mObj, ax, grid_type='grid2', map_type='map2')
mapDatObj.overlayMapFitVel()
mapDatObj.overlayCnvCntrs()
mapDatObj.overlayHMB()

savefig('pltMapGrd_test2.png')

results in:

Since theses are new filetypes, there are limited amounts of either on the sd-data.ece.vt.edu server. Grid2/map2 files are available for mostly of 2017 and fitacf3 files are available from 20171201 to near present (have had to revert to rst4.0 to catch up on old data type reprocessing).

[egthomas]

@ksterne - are you sure davitpy is plotting the potential contours correctly? The attached figure is what I get from the Map Potential Plot tool on the VT website (http://vt.superdarn.org/tiki-index.php?page=DaViT+Map+Potential+Plot) for a "map2" file:

[ksterne]

I cannot not speak to how well davitpy is doing the mapping. This PR is only concerning being able to access the new data filenames. Updates to any of the plotting tools was more so to expose those tools to the updated code in sdio/. Of which I only really updated the documentation and nothing in the code itself.

As well, the code used to generate the map was just pulled from the plotMapGrd.MapConv class example. I only listed it as a way to try to access grid and/or map data. It's entirely possible that this code is wrong or out of date. I can remove the example listed here if it's a distraction and just leave the test up to the sdDataTypes.py code.

[egthomas]

@ksterne no worries, I was just curious to see how the new map data looked. It could just be a matter of the number of contours being fixed rather than only drawing contours at fixed intervals.

[aburrell]

I ran the code and downloaded the files from VT, and while the first figure looks good the map is not the same.

[ksterne]

Were you able to get matching file checksums from what I put into the sdDataTypes.py file? This could be an issue with the plotting code and not the file. Maybe run another checksum on the resulting map2 file that should in up in your cached file directory if send over what checksum algorithm you use (sha1sum maybe?) and I can check the one I'm getting against that?

[aburrell]

I'm not sure what you mean by "matching file checksums"

[ksterne]

I mean the tests that happen whenever you run python sdDataTypes.py . I updated the code that's executed so that the expected checksum should match the checksum for the file that gets transferred to your computer.

[ksterne]

Hey @aburrell, were you able to run the code as I mentioned here? I know it's little fundamental, but from the directory the file's in, it's just a command line run of python sdDataTypes.py.

[aburrell]

It's not been simple to do, since the code only works if your davitpyrc is set up to default to VT.

[aburrell]

So, if I run: vtptr = davitpy.pydarn.sdio.sdDataPtr(sdate, "north", "map2") I get file: 20170403.040000.20170404.040000.north.map2 If I run the map command I get file: 20170403.040000.20170403.040200.north.map2

Running os.stat and hashlib on the first file I get: File size: 29260764 Md5sum: 490e50b97b1e0d5c3e068530aed8db82

Running the same routines on the second file I get: File size: 11757614 Md5sum: 743534fe7eda6ba53c884fe72f179ddc

[ksterne]

But how did you get map2 file if the code isn't looking at the VT server? I guess you could have downloaded RST 4.1, converted processing scripts, and processed new grid2, map2 files...but that's weirdly too coincidental. I do see a problem of having test code based on VT files when that's not the only place the files could exist.

I think you've got something a little backwards, or at least it's not clear on your results of what the "first file" and the "second file" are. Do you mean back to my original test or to the two files you listed (20170403.040000.20170404.040000.north.map2, and 20170403.040000.20170403.040200.north.map2)? Both of the map2 files you just listed (the second choice in my question) should have the same file size and thus md5sum. I am see these match with the "second file" you've listed numbers/hash for. But I'm not sure where you got the figures for the "first file".

[aburrell]

It’s easy enough for me to change the rcparams in ipython to VT, but not easy to do when running from the command line. That’s the problem with having command line tests that assume a particular rcparam setup. If tests for the command line scripts are going to use VT as a default (which is appropriate), then this information should be included as strings instead of using rcParams.

By first and second file, I mean the first and second file I listed. The test for the pull request gave me the first file, and the test code in the script you asked me to run gave me the second. If these files are supposed to be the same, then there’s a problem because they aren’t.

On 14 Mar 2018, at 09:43, Kevin Sterne notifications@github.com wrote:

But how did you get map2 file if the code isn't looking at the VT server? I guess you could have downloaded RST 4.1, converted processing scripts, and processed new grid2, map2 files...but that's weirdly too coincidental. I do see a problem of having test code based on VT files when that's not the only place the files could exist.

I think you've got something a little backwards, or at least it's not clear on your results of what the "first file" and the "second file" are. Do you mean back to my original test or to the two files you listed (20170403.040000.20170404.040000.north.map2, and 20170403.040000.20170403.040200.north.map2)? Both of the map2 files you just listed (the second choice in my question) should have the same file size and thus md5sum. I am see these match with the "second file" you've listed numbers/hash for. But I'm not sure where you got the figures for the "first file".

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/pull/341#issuecomment-373046002, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuC_mKYa1ZLyuRyRGhMX7rZwXo355V6ks5teSyJgaJpZM4SVNqw.

[ksterne]

Hey @aburrell, I think I get it now. You're wondering if the file 20170403.040000.20170404.040000.north.map2 should match the file 20170403.040000.20170403.040200.north.map2? I wouldn't think so since by the names mean they should contain different amounts of data.

That makes sense on not relying on a particular rcparams setup and instead hardcoding in the setup. Though that creates some burden in the future if things change around (I hope they don't!). I've pushed some commits up here that might help you test as they now not relying on VT-specific rcParam configs. Mind giving it a test here to see if the last two in the test code make sense?

I should also add I caught two things that needed attention in the documentation for sdDataPtr as there wasn't anything listed for remote_site and the port variable apparently needs to be a string instead of an integer.

[egthomas]

It looks like the velocity vector magnitudes agree much more closely between the IDL plot from the VT website I posted and @aburrell 's figure compared to @ksterne 's original convection map plot. Particularly near 80 degrees MLAT between 17-20 MLT (north of Inuvik) and just northeast of Svalbard.

If I choose the mapex option on the VT website instead of map2, I get a figure much closer to @ksterne 's :

[ksterne]

Mmm, interesting. I just ran this again on (probably) another system and I got the following plot. So, I'm guessing I mixed up the images in my original post and instead that one shows data for mapex instead of map2 ( or grid2). Again, I think the better test here is within the test codes for radDataTypes.py and sdDataTypes.py. I was just showing the plots to show that there was access to the new data types. Though it is good to know that it's accessing the new versions instead of the old versions of grid and map files.

Let me know what you find with testing the code after the commits I made @aburrell.

[aburrell]

Ok, running tests. Started with the command line test and got the stuff below. It looks like it worked. I hear you about maintaining the set-up, but I think that will be less of an issue when we move to unit tests and can have all of that in one place.

Output:

$ python sdDataTypes.py Expected File: /tmp/sd/20120710.000000.20120711.020000.north.mapex

Running sftp grab example for sdDataPtr. Environment variables used: DB: sd-data.ece.vt.edu DB_PORT: 22 DBREADUSER: sd_dbread DBREADPASS: 5d DAVIT_SD_REMOTE_DIRFORMAT: data/{year}/{ftype}/{hemi}/ DAVIT_SD_REMOTE_FNAMEFMT: {date}.{hemi}.{ftype} DAVIT_SD_REMOTE_TIMEINC: 24 DAVIT_TMPDIR: /tmp/sd/ Actual File Size: 33008910 Expected File Size: 33008910 Actual Md5sum: 1656c94ca564c9a96821496397eed037 Expected Md5sum: 1656c94ca564c9a96821496397eed037 Let's read two records from the remote sftp server: 1341878520.0 1341878640.0 Close pointer reopen pointer Should now be back at beginning: 1341878520.0 What is the current offset: 18092 Try to seek to offset 4, shouldn't work: 20480 What is the current offset: 20480

Running local grab example for sdDataPtr. Environment variables used: DAVIT_SD_LOCAL_DIRFORMAT: /sd-data/{year}/{ftype}/{hemi}/ DAVIT_SD_LOCAL_FNAMEFMT: {date}.{hemi}.{ftype} DAVIT_SD_LOCAL_TIMEINC: 24 DAVIT_TMPDIR: /tmp/sd/ Error: Failed to create expected cache file Let's read two records: ERROR:root:the pointer does not point to any data record read failed for some reason

Now lets grab an RST4.1 and later map2 file type Expected File: /tmp/sd/20170710.000000.20170711.020000.south.map2

Running sftp grab example for sdDataPtr. Environment variables used: DB: sd-data.ece.vt.edu DB_PORT: 22 DBREADUSER: sd_dbread DBREADPASS: 5d DAVIT_SD_REMOTE_DIRFORMAT: data/{year}/{ftype}/{hemi}/ DAVIT_SD_REMOTE_FNAMEFMT: {date}.{hemi}.{ftype} DAVIT_SD_REMOTE_TIMEINC: 24 DAVIT_TMPDIR: /tmp/sd/ Actual File Size: 28284376 Expected File Size: 28284376 Actual Md5sum: de91b6bc239e0ff069732b1ecba5ecf1 Expected Md5sum: de91b6bc239e0ff069732b1ecba5ecf1

Now lets grab an RST4.1 and later grid2 file type Expected File: /tmp/sd/20170710.000000.20170711.020000.north.grid2

Running sftp grab example for sdDataPtr. Environment variables used: DB: sd-data.ece.vt.edu DB_PORT: 22 DBREADUSER: sd_dbread DBREADPASS: 5d DAVIT_SD_REMOTE_DIRFORMAT: data/{year}/{ftype}/{hemi}/ DAVIT_SD_REMOTE_FNAMEFMT: {date}.{hemi}.{ftype} DAVIT_SD_REMOTE_TIMEINC: 24 DAVIT_TMPDIR: /tmp/sd/ Actual File Size: 11931978 Expected File Size: 11931978 Actual Md5sum: c7f555249fc18244f61bb118cc71b2e1 Expected Md5sum: c7f555249fc18244f61bb118cc71b2e1

[aburrell]

And here are the plots from the original tests:

[aburrell]

So with @ksterne getting the same thing as me from a different system, I'm ok with this now.

[ksterne]

Thanks @aburrell for sticking with this and in general making the test here better! Anyone else feel like giving it a test out? Maybe @egthomas?

[egthomas]

No testing here, just a general suggestion that for plotting the potential contours should be changed to specific levels (ie something like +/- 3 kV, 9 kV, 15 kV, etc) rather than a fixed number of 12 linearly spaced contours regardless of the total cross polar cap potential drop.

edit and yes I know that's outside the scope of the pull request

[ksterne]

Sounds good @egthomas, I'm betting that's a better as a feature request in the Issues to add as a to-do item for us than buried in this pull request. But, am I reading that you aren't going to test this pull request?

[aburrell]

@ksterne , @egthomas doesn't use python yet, so probably not the right person to test. @Shirling-VT , would you be willing to test this?

[egthomas]

@ksterne , @egthomas doesn't use python yet

good one @aburrell !

[aburrell]

We'll get you there eventually!

[ksterne]

I agree! One day @egthomas.

I'm in need of someone to give it a second test here then, @Shirling-VT, @MuhammadVT, or even @asreimer?

[Shirling-VT]

Below are the plots I get from the testing codes:

They look the same with @aburrell testing results, good to merge?

[asreimer]

I tested this by running:

python radDataTypes.py
python sdDataTypes.py

and everything worked as expected. I ran the code @ksterne provided and get the same RTI as @Shirling-VT, @aburrell, and @ksterne. Also, same plot when I run the code to make the map plot.

Reviewing the source code changes also looks good to me. Merging.