RELEASE: pyDARN Version 4.0!

[carleyjmartin]

pyDARN Version 4.0

Deadline: 21st December 2023

Important Notes:

• Please read this top comment in full before testing to avoid common issues!
• pyDARN has an updated requirement for Python 3.7, pyDARN will not install for python 3.6!

Scope

This major release includes:

• NEW IQ level data plotting
• NEW Latitude and longitude y-axis in RTP
• NEW Ball and stick plots
• NEW Map file variable time series plotting
• NEW Terminator plotting
• All plots now return the same dictionary
• Coastlines available in magnetic coordinate spatial plots without Cartopy
• More flexibility in fan plots - single beams/ range gate range options
• TDiff correction for elevation data available
• Boxcar filtering available for data before plotting
• Corrections to geolocation algorithms
• Standardized plot return values across all plots
• Updated GSMR range estimation
• Bug fix Radar position labels no longer overlap
• Bug fix Warning use refactored
• Bug fix Multiple bug fixes for data handling and plotting in grid plots
• Bug fix Multiple bug fixes for the aesthetics of convection maps

Some of which are yet to be merged in so please go check out the active PR's first! #307 #338 #343 #353 #355 #356

I have ordered the author list in the zenodo by contribution, developers first, then testers, then previous large contributors to the project in alphabetical order, let me know if there is anyone missing!

Approval

Number of approvals: 3 code review and testing

Test

Testing Checklist (feel free to tick off what you do so at least one person has tested everything):

• [x] Are your details in the .zenodo.json file correct and are you happy with your coauthor position (email Carley if you are missing!)?

• [x] Make sure the library installs properly using pip3 install git+https://github.com/superdarn/pydarn@release/4.0

• [x] Install by git clone and use pytest - process is below (this only tests that the methods work without error, this does not test that the output of the methods are scientifically correct - that's where you come in ;) ).

• [x] Read the documentation (link below), is there anything that wasn't helpful, missing or could be explained better?

• [x] Code review - read the lines of code changed (Files Changed tab above), do they make sense, are they optimal?

• [x] Generally test the following methods using as many keyword combos as you can (use documentation linked below to write code):

  • [x] Summary plots: pydarn.RTP.plot_summary()
  • [x] Range time plots: pydarn.RTP.plot_range_time()
  • [x] Time series plots: pydarn.RTP.plot_time_series()
  • [x] Grid plots: pydarn.Grid.plot_grid()
  • [x] Convection maps: pydarn.Maps.plot_mapdata()
  • [x] ACF plots: pydarn.ACF.plot_acfs()
  • [x] Check that outputs from the plotting features are in the format:{'ax': ax, 'ccrs': ccrs, 'cm': cmap, 'cb': cb, 'fig': plt.gcf(), 'data': {}} Where data contains any interesting plot data you might like to use.

• [x] Specific new features that need testing (should also be in documentation and see linked PRs for testing info but make sure you test on this release branch, they have already been tested individually):

  • [x] Time series plots for maps #336
  • [x] Coordinate-time plots #331
  • [x] TDiff recalculation for elevation #290
  • [x] Plotting single beams on fan plots #298
  • [x] Boxcar filtering for all plot types #296
  • [x] Ball and Stick plots #304
  • [x] Plotting a terminator #303
  • [x] Using the RangeEstimation.GSMR_BRISTOW and updated GSMR
  • [x] IQ Plotting #338

• You may need to amend any old code to work - this is expected and probably not a bug!

• Please comment to say everything worked (if so please tick the approve box on Files Changed - > Review Changes -> Approve + type any comments in box -> Submit Review).

• If there are errors please comment your code used, the traceback and any errors/warnings in the command line, and any output plots if any.

• For approvals or errors, please specify your OS, python version, matplotlib version and installation process. Thanks :)

• I will likely add to this when newer PRs are merged in later.

Please tick if you have successfully tested using:

• [x] MacOS
• [x] Windows
• [x] RPM distro (Opensuse, Fedora +)
• [x] Debian distro (Ubuntu +)

Here are some misc data files that you can use to test with: fitacf.zip maps_n_grids.zip Rawacf is too big for github but you can get any off FRDR

Additional little things

Read the docs link will appear here when I've done it: https://pydarn.readthedocs.io/en/release-4.0/

pytest modules have been updated, you can run pytest by cloning the repository, moving to this release branch, installing the branch, installing pytest and running pytest (this may take 5-10 minutes to run all tests):

git clone https://github.com/SuperDARN/pydarn.git
cd pydarn
git checkout release/4.0
pip3 install .
pip3 install pytest
pytest

As always, if you need help installing and testing, feel free to give me an email.

Thanks for everyone's help this year on this release! Carley

[hiyadutuje]

Hello Carley, Good job! What do you want me to do? I don't undersand what you mean by "assigned #358 to me". Best Regards,Alicreance On Monday, 13 November 2023 at 18:47:34 GMT+2, Carley @.***> wrote:

Carley @.***) is not on your Guest List | Approve sender | Approve domain

Assigned #358 to @hiyadutuje.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were assigned.Message ID: @.***>

[hiyadutuje]

Dear All,

I have done the test. This is what I get from my linux computer:

"platform linux -- Python 3.9.17, pytest-7.4.0, pluggy-1.0.0 rootdir: /home/alicreance/pydarn plugins: anyio-3.5.0 collected 611 items

test/test_Fan.py ................................................ [ 7%] test/test_Grid.py ..................... [ 11%] test/test_RTP.py ....................................................... [ 20%] ........................................................................ [ 32%] ........................................................................ [ 43%] ........................................................................ [ 55%] ........................................................................ [ 67%] ........................................................................ [ 79%] ........................................................................ [ 90%] ............................................ [ 98%] test/test_maps.py ........... [100%]

============================= 611 passed in 42.62s =============================

"

[hiyadutuje]

I am not getting an error after unzipping the files, but the plot is empty for all the files I have tried. Not sure why!

[carleyjmartin]

@hiyadutuje Thanks for all the testing! You can email me some of the map files that's not plotting anything above, I suspect it might just be that the field is empty in the file, otherwise it would give a field not found error, but I can look into it more. The y-axis looks very small, it could be that there is something plotted but outside those bounds, if you can add your code used the the email too that would be helpful!

The other tests (on other PRs) seem to output what I expect, only a few warnings are not showing up so I can fix them for the release to make more sense.

[PrestonXPitzer]

Map time series plotting is working for me on this version using code

import bz2
import pydarn
import datetime as dt
import matplotlib.pyplot as plt

file_path = "20170113.n.map"
data = pydarn.SuperDARNRead().read_dmap(file_path)
pydarn.Maps.plot_time_series(data, parameter=pydarn.Maps.TimeSeriesParams.NUM_VECTORS, start_record=0, end_record=100)
plt.show()

I also had a successful run of pytest on windows

[Doreban]

Successfully installed on Windows 10 and OpenSUSE 15.4 after updating from python 3.6 to 3.11 and installing python 3.11 development tools.

Tested some plotting functions on windows using python 3.9.2 and matplotlib version 3.8.2

Summary plots are working. I also tested changing the plotting options in the function, but here is a basic summary plot. made using:

fitacf_file = "20230808.1406.00.pgr.a.fitacf"
darn_read = pydarn.SuperDARNRead(fitacf_file)
fitacf_data = darn_read.read_fitacf()
pydarn.RTP.plot_summary(fitacf_data, beam_num=2, range_estimation=pydarn.RangeEstimation.RANGE_GATE)

I noted that in the plot_summary() comment block that the option for line_color was not described.

Range time plots also appear to work. made using the code below, I was also playing with changing the range estimation used. :

fitacf_file = "20230808.1406.00.pgr.a.fitacf"
darn_read = pydarn.SuperDARNRead(fitacf_file)
fitacf_data = darn_read.read_fitacf()
pydarn.RTP.plot_range_time(fitacf_data, beam_num=2, groundscatter=True, background="pink",
                               range_estimation=pydarn.RangeEstimation.GSMR_BRISTOW, parameter='elv')

I also noted in the plot_range_time() comment block that the the option labelled filter_settings is named plot_filter in the comment.

Plotting the lag0 power worked well as well. I included an arbitrary filter just to check if the filters worked. made using:

from matplotlib import colors

rawacf_file = "20231105.1000.00.inv.a.rawacf"
darn_read = pydarn.SuperDARNRead(rawacf_file)
rawacf_data = darn_read.read_rawacf()
lognorm = colors.LogNorm
filts = {'min_array_filter': {'pwr0': 10}}
pydarn.RTP.plot_range_time(rawacf_data, beam_num=2, parameter='pwr0', norm=lognorm, cmap='gnuplot',
                               filter_settings=filts)

[Shirling-VT]

MacOS Monterey 12.5.1 matplotlib 3.7.3 python 3.8.0

pytest passed:

I used the below code to test the grid plotting, the magnetic coordinate works but the geographic coordinate does not work:

import pydarn
import datetime as dt
import matplotlib.pyplot as plt

grid_north = "/Users/xueling/data/superdarn/2015/grid2/north/20150317.north.grid2"
grid_data = pydarn.SuperDARNRead(grid_north).read_grid()

gridplot = pydarn.Grid.plot_grid(grid_data, record=5, parameter = 'vel',fov_color='grey',
                                 lowlat=30, coastline=True, coords=pydarn.Coords.GEOGRAPHIC,
                                 projs=pydarn.Projs.GEO)
plt.show()

gridplot = pydarn.Grid.plot_grid(grid_data, record=5, parameter = 'vel', fov_color='grey',
                                 lowlat=30, coastline=True)
plt.show()

Looks like the "end_g_thetas" is not defined before using:

[Shirling-VT]

Time series and ACF plots are working fine, thus ticked both:

# Time series plot
#Read in fitACF file
pyk_file = '/Users/xueling/data/superdarn/2015/fitacf/pyk/20150308.1401.00.pyk.fitacf.bz2'
pyk_data = pydarn.SuperDARNRead().read_dmap(pyk_file)

a = pydarn.RTP.plot_time_series(pyk_data, beam_num=3, parameter='v', gate=20)
plt.show()

b = pydarn.RTP.plot_time_series(pyk_data, parameter='cp', date_fmt=('%H:%M'), beam_no=7)
plt.show()

# ACF plot
file = "/Users/xueling/data/superdarn/2014/rawacf/rkn/20140105.1208.03.rkn.rawacf.bz2"
rawacf_data =  pydarn.SuperDARNRead().read_dmap(file)

#plt.figure(figsize=(10, 6))
a = pydarn.ACF.plot_acfs(rawacf_data, beam_num=7, gate_num=44, normalized=False,
                         start_time=dt.datetime(2014, 1, 5, 12, 8), pwr_and_phs=True,
                         plot_blank=True)
plt.show()

b = pydarn.ACF.plot_acfs(rawacf_data, beam_num=15, gate_num=16, start_time=dt.datetime(2014, 1, 5, 13, 30), pwr_and_phs=False)
plt.show()

[Shirling-VT]

Tested the terminator, it's working well using the code below, although we need to update the old code to find the new_coordinate for the last plot. The read doc for coordinate and coordinates.md need to be updated as well to find the correct location of new_coordinate (or import it at the beginning): https://pydarn.readthedocs.io/en/release-4.0/user/coordinates/

################################# Terminator plot#############################
# Geographic Coordinates
# North Winter
a=pydarn.Fan.plot_fov(66, dt.datetime(2023, 12, 21, 0, 0), lowlat= 10,
                      coords=pydarn.Coords.GEOGRAPHIC ,projs=pydarn.Projs.GEO,
                      boundary=True, line_color='red', coastline=True, nightshade=250)
plt.show()
# 
# North Summer
a=pydarn.Fan.plot_fov(66, dt.datetime(2023, 6, 21, 0, 0), lowlat= 10,
                                  coords=pydarn.Coords.GEOGRAPHIC ,projs=pydarn.Projs.GEO,
                                  boundary=True, line_color='red', coastline=True, nightshade=250)
plt.show()

# South Summer
a=pydarn.Fan.plot_fov(24, dt.datetime(2023, 12, 21, 0, 0), lowlat= 10,
     coords=pydarn.Coords.GEOGRAPHIC ,projs=pydarn.Projs.GEO,
     boundary=True, line_color='red', coastline=True, nightshade=250)
plt.show()

# South Winter
a=pydarn.Fan.plot_fov(24, dt.datetime(2023, 6, 21, 0, 0), lowlat= 10,
     coords=pydarn.Coords.GEOGRAPHIC ,projs=pydarn.Projs.GEO,
     boundary=True, line_color='red', coastline=True, nightshade=250)
plt.show()

# Magnetic Coordinates to show warning
# North Winter
a=pydarn.Fan.plot_fov(66, dt.datetime(2023, 12, 21, 0, 0), lowlat= 5,
    boundary=True, line_color='red', coastline=True, nightshade=250)

# test to plot terminator if ever required - plot not fill!
# Get antisolar point in geographic coords and radius of terminator
# at given height
date = dt.datetime(2023, 12, 21, 0, 0)
antisolarpsn, arc, ang = pydarn.terminator(date, 250)
# Convert position to magnetic coordinates
mlat, lon_mag, _ =  aacgmv2.convert_latlon(antisolarpsn[1],
                                                 antisolarpsn[0],
                                                 250,
                                                 date,
                                                 method_code='G2A')
# Shift to MLT
shifted_mlts = lon_mag - (aacgmv2.convert_mlt(lon_mag, date) * 15)
shifted_lons = lon_mag - shifted_mlts
mlon = np.radians(shifted_lons)
# Get positions at a distance from new position to plot terminator
lats = []
lons = []

for b in range(-180,180,1):
    (lat, lon) = pydarn.utils.general_utils.GeneralUtils.new_coordinate(mlat, shifted_lons, arc, b, R=pydarn.Re)
    nlon =np.radians(lon)
    lats.append(lat)
    lons.append(nlon)
lats2 = np.zeros(len(lats))
plt.plot(np.squeeze(lons), np.squeeze(lats), color='b', zorder=2.0, linewidth=6.0, linestyle='dashed')

plt.show()

[carleyjmartin]

Thanks for testing everyone! I think we've got most of the bugs sorted out, I'm going to spend a day doing some testing and read over and amend the docs. I expect that I won't be able to do the release by tomorrow, so I'll schedule some time early Jan to do the release, and then I'll be available to field any questions about code changes with the new version (rather than me do the release then go on vacation for a week).