Develop protocol to read ascii gridded data

[TaraJensen]

Not sure what the protocol should look like. I'm guessing for now it might be a python script to read the data.

From Lutz Rastaetter Mon, Jun 10, 1:23 PM Here is a sample ASCII file from the CalcDeltaB model generated by the online visualization:

https://ccmc.gsfc.nasa.gov/idl_images/pointdata_201288084585_20050514_224000.txt NOTE: did not pull down data in time so have requested it be reposted - will attach when received

Data format string: '(6E12.4)' Data printout from CCMC-simulation: version PD-2.0 Data type: CalcDeltaB-grid magnetosphere Run name: Lutz_Rastaetter_20190520_PP_1 Model coordinate system: MAG Missing data: NaN Date, time: 2017 5 28 10:00:00 Output data: field with 73x36=2628 elements mlon mlat r B_north B_east B_down
deg. deg. R_E [nT] [nT] [nT]

0.0000E+00 -8.7500E+01 1.0000E+00 2.4309E+01 -2.6194E+02 -3.8589E+02

5.0000E+00 -8.7500E+01 1.0000E+00 7.9700E-02 -2.6504E+02 -3.7958E+02

1.0000E+01 -8.7500E+01 1.0000E+00 -2.4018E+01 -2.6575E+02 -3.7366E+02

1.5000E+01 -8.7500E+01 1.0000E+00 -4.7803E+01 -2.6418E+02 -3.6819E+02

2.0000E+01 -8.7500E+01 1.0000E+00 -7.1121E+01 -2.6046E+02 -3.6322E+02

2.5000E+01 -8.7500E+01 1.0000E+00 -9.3827E+01 -2.5470E+02 -3.5874E+02

…

All header lines start with “# “ and contain these metadata:

“Data format string: “ is followed by the Fortran format string that may be used to read the data (lines following the header),

“Data Type:” is followed by the model name (“CalcDeltaB-grid”) and the domain (“magnetosphere”),

“Run name:” followed by the simulation run name

“Model Coordinate system:” followed by “MAG” (or “GEO” for this model).

“Date, Time:” is followed by the year, month and day numbers and the UT time (formatted as hh:mm:ss),

“Missing Data: is followed by the value that is deemed missing or NaN,

“Output data:” followed by “field with 73x36 elements” indicates the 2D shape of the output.

                        Here there are 73 longitudes (varying first) and 36 latitudes (varying slowest).

The last two lines of the header are the names of the coordinates (mlon mlat r) and output quantities (B_north B_east B_down), followed by a line with the respective units (sometimes enclosed in “[“ and “]”). Dimensionless units should always be indicated by square brackets enclosing nothing (“[]”).

[TaraJensen]

Charge 270001

[TaraJensen]

Sample file: pointdata_883280022096_20160809_082000.txt

Resulting plot:

[rgbullock]

I took the sample data file from Tara's June 18th note and wrote a C++ program to read the point data and plot it (see attached png file "raw.png"). I arbitrarily chose the north component of magnetic field as the "data" values. I then wrote a python3 script to read the sample file, load the point obs, and write them out to a piclke file. I was then able to read the pickle file with the C++ python embedding code and do a plot (attached file "pickle.png"). I've also attached a copy of the python script that reads the ascii data file and write pickle output ("do_pickle.py")

do_pickle.py.pdf

Not sure what remains to be done here except perhaps formalizing the python representation of point data.

[JohnHalleyGotway]

Reassigning to George to finalize this example, make sure that it works via plot_data_plane, and add another example to the website: https://dtcenter.org/community-code/model-evaluation-tools-met/sample-analysis-scripts

[georgemccabe]

Here is the modified script to read the input file, pull out the time, grid information, and data. It sets met_input, which is a dictionary that contains a key 'met_data' which contains the numpy array stored in met_data, and the key 'attrs' which contains the attributes dictionary stored in attrs.

read_CDB_ascii.py.txt

[georgemccabe]

An example calling plot_data_plane using this script (with input data and output ps file) are now on the sample analysis script webpage listed above.

[georgemccabe]

While at NASA Goddard, we learned that the script should be able to convert geomagnetic lat/lon values to geographic lat/lon if plotting. I updated the script to have an optional argument 'geog' at the end of the command that will convert the values based on the valid time of the data using the geopack python library. I updated the example on the sample scripts webpage to show the new command, new script, and new output file (replaced the .ps file with a .png so you can click and view it easily). This script should be called without the 'geog' option if comparing the data to other data that is on geomagnetic coordinates.