ESMF for creating a new post-processing package

[twfang]

To make sure the grid transformation between irregular grid and uniform grid in IPE is done correctly, we need to develop the new pos-processing package taking advantage of ESMF. In the irregular IPE grid, each point has given a geographic longitude/latitude. The irregular grid needs to be mapped onto a uniform 3-D grid. For now, let's use what we have currently. I assume it is easy to be changed later. We need to eventually increase the altitude to at least 2000 km with a reasonable height resolution, which can be defined later.

Latitude: -90:2:90 (91 total, 2 degree interval) Logitude: 0:4:356 (90 total, 4 degree interval) Altitude: 90:5:1000 (183 total, 5 km interval)

[twfang]

1. Basic requirements: TEC (vs. GloTEC), NmF2, hmF2, O/N2 (vs. GOLD observation), Neutral density at 400 km bring back to SWPC. a. For CONOPS 1 (with WAM-IAU, and 2-day forecast), hourly output b. For CONOPS 2 (nowcast with real-time drivers), 3 min output

2. Other necessary fields to save on WCOSS: 3-D winds, neutral species densities, Tn, mean mass, all ions, Ti, Te with variable height grid, ExB velocity, saved in every 15 minutes and stored at HPSS.

[twfang]

After our discussion today, two IO layers will be established.

1. IO with access to WAM that outputs neutral parameters in pressure level, latitude, longitude
2. IO with access to IPE that outputs all neutral and plasma parameters in height, latitude, longitude

[twfang]

Here are the parameter names that we need for output fields from IPE,

3-D winds: ipe % neutrals % velocity_geographic

neutral species densities: ipe % neutrals % oxygen ipe % neutrals % hydrogen ipe % neutrals % helium ipe % neutrals % nitrogen ipe % neutrals % molecular_oxygen ipe % neutrals % molecular_nitrogen

Tn: ipe % neutrals % temperature

all ions: ipe % plasma % ion_densities

1. O+, 2. H+, 3. He+, 4. N+, 5. NO+, 6. O2+, 7. N2+, 8. O+(2D), 9. O+ (2P)

Ti: ipe % plasma % ion_temperature

Te: ipe % plasma % electron_temperature

ExB velocity: ipe % eldyn % v_ExB_apex

Mean Mass: Undecided yet, perhaps need to come from WAM directly.

[twfang]

Some information for using COMmon I/O from Raffaele.

I've pushed to GitHub an initial revision of the COMmon I/O (COMIO) library, which I will use to build the Space Weather I/O (SWIO) component. This library provides a common Fortran interface (APIs) to read/write data arrays to file using either HDF5 or NetCDF (PnetCDF) format.

I've simplified the APIs as much as possible, so that the only steps required to read/write a file can be summarized as follows:

use mpi use comio

integer :: ierr, fmt class(COMIO_T), pointer :: io => null()

call mpi_init(ierr) ! To initialize COMIO (once) ! fmt = COMIO_FMT_HDF5 to select HDF5 I/O, or ! fmt = COMIO_FMT_PNETCDF to select PnetCDF I/O ! comm = MPI communicator ! info = MPI info object (set to MPI_INFO_NULL most of the time) io => COMIO_T(fmt=fmt, comm=MPI_COMM_WORLD, info=MPI_INFO_NULL) ! open file as: create ("c"), write ("w"), or read ("r") call io % open(filename, "c") ! Set domain decomposition ! global domain: global = (/ NX, NY /) ! local decomposition: start(:,:) and count(:,:) arrays (as for NetCDF) ! Note that data(:,:) holds local data, so it is dimensioned as: ! data(count(1),count(2)) call io % domain(global, start, count) call io % write(name, data) ! use io % read(name, data) to read in data ! repeat above sequence as needed: ! call io % domain() if different ! call io % write(name1, data1) call io % close() ! To shut down COMIO (at the end of all I/O operations) call io % shutdown() call mpi_finalize(ierr)

You may download the code as:

git clone https://github.com/NOAA-SWPC/COMIO.git

Then build it (on Hera) as:

module load intel/18.0.5.274 module load impi/2018.0.4 module load hdf5parallel/1.10.5 module load pnetcdf/1.10.0 module load szip/2.1

./configure make make check

Note that COMIO can be configured with support for either HDF5 or PnetCDF or both via configure flags (run ./configure --help for details). For example:

./configure --with-hdf5 --without-pnetcdf

[twfang]

I have received some preliminary files from the new IO component provided by Raffaele. Running the same version of the code, here are some comparisons of results from old and new methods.

From master branch at 03160057 at 150 km

at 400 km

The fluctuations we see from the new method (bottom) don't look right. They seem to follow the fluctuations of the background magnetic field.

[gmillward]

Do we know how ESMF interpolates between grids - I'd assumed that they had nailed this problem - but I guess it could be similar to what we currently have.

My solution was to find the three nearest points (on flux-tubes) to a given geographic location and - their distances - and then do a weighted average

• all offline of course (it takes ages) - and then we have all of these integer and factor arrays to provide the answer.

I thought my solution was 'clunky' - but it worked - just wondering what underpins the ESMF solution?

On Mon, Feb 24, 2020 at 1:25 PM twfang notifications@github.com wrote:

I have received some preliminary files from the new IO component provided by Raffaele. Running the same version of the code, here are some comparisons of results from old and new methods.

From master branch at 03160057 at 150 km [image: 150km] https://user-images.githubusercontent.com/22968399/75188177-e7e32780-5708-11ea-9cd5-104de0b1d23d.png

at 400 km [image: 400km] https://user-images.githubusercontent.com/22968399/75188195-ec0f4500-5708-11ea-9e26-1e3ef78879af.png

The fluctuations we see from the new method (bottom) don't look right. They seem to follow the fluctuations of the background magnetic field.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NOAA-SWPC/IPE/issues/7?email_source=notifications&email_token=AF2U5HSIMVHUOWWDNLVMHMDREQUKDA5CNFSM4JRO7DPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMZM2FA#issuecomment-590531860, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF2U5HUBIZUUCQARTYJZZXDREQUKDANCNFSM4JRO7DPA .

--

Dr. George Millward Cooperative Institute for Research in Environmental Sciences (CIRES) University of Colorado, Boulder

Visiting scientist at NOAA Space Weather Prediction Center Boulder, CO

george.millward@noaa.gov

[timfullerrowell]

They did give us a description earlier, about how they define a mesh etc. It uses the same method, I assume, as the previous WAM to IPE regridding, but in the other direction. This is probably just a bug in the regridding code from IPE to WAM, which we hadn't fully validated before.

On Mon, Feb 24, 2020 at 1:59 PM gmillward notifications@github.com wrote:

Do we know how ESMF interpolates between grids - I'd assumed that they had nailed this problem - but I guess it could be similar to what we currently have.

My solution was to find the three nearest points (on flux-tubes) to a given geographic location and - their distances - and then do a weighted average

• all offline of course (it takes ages) - and then we have all of these integer and factor arrays to provide the answer.

I thought my solution was 'clunky' - but it worked - just wondering what underpins the ESMF solution?

On Mon, Feb 24, 2020 at 1:25 PM twfang notifications@github.com wrote:

I have received some preliminary files from the new IO component provided by Raffaele. Running the same version of the code, here are some comparisons of results from old and new methods.

From master branch at 03160057 at 150 km [image: 150km] < https://user-images.githubusercontent.com/22968399/75188177-e7e32780-5708-11ea-9cd5-104de0b1d23d.png

at 400 km [image: 400km] < https://user-images.githubusercontent.com/22968399/75188195-ec0f4500-5708-11ea-9e26-1e3ef78879af.png

The fluctuations we see from the new method (bottom) don't look right. They seem to follow the fluctuations of the background magnetic field.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub < https://github.com/NOAA-SWPC/IPE/issues/7?email_source=notifications&email_token=AF2U5HSIMVHUOWWDNLVMHMDREQUKDA5CNFSM4JRO7DPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMZM2FA#issuecomment-590531860 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AF2U5HUBIZUUCQARTYJZZXDREQUKDANCNFSM4JRO7DPA

.

--

Dr. George Millward Cooperative Institute for Research in Environmental Sciences (CIRES) University of Colorado, Boulder

Visiting scientist at NOAA Space Weather Prediction Center Boulder, CO

george.millward@noaa.gov

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NOAA-SWPC/IPE/issues/7?email_source=notifications&email_token=AH5BFOGZUFKEUOXTMFVW5V3REQYK7A5CNFSM4JRO7DPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMZQFPQ#issuecomment-590545598, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH5BFOGHNOOYI7BR6UTYJVDREQYK7ANCNFSM4JRO7DPA .

[rmontuoro]

SWIO does not perform regridding, but relies on regridding carried out by standard NUOPC connectors to increase flexibility. Such regridding uses standard ESMF methods and is unrelated to the WAM->IPE regridding method implemented in the SWPC mediator.

The provided output files were generated by using default ESMF bilinear regridding. No cell average is performed in this method. I believe this may be the reason why features related to IPE's flux tubes appear in the output values. We can investigate other regridding methods next.

[gmillward]

Raffaele

Do you know the references/links for this ESMF bilinear regridding (and related stuff I guess) - would be very interested to see how this works.

Cheers

George.

On Tue, Feb 25, 2020 at 11:48 AM rmontuoro notifications@github.com wrote:

SWIO does not perform regridding, but relies on regridding carried out by standard NUOPC connectors to increase flexibility. Such regridding uses standard ESMF methods and is unrelated to the WAM->IPE regridding method implemented in the SWPC mediator.

The provided output files were generated by using default ESMF bilinear regridding. No cell average is performed in this method. I believe this may be the reason why features related to IPE's flux tubes appear in the output values. We can investigate other regridding methods next.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NOAA-SWPC/IPE/issues/7?email_source=notifications&email_token=AF2U5HRTB3OQAWMLHS4RINDREVRYNA5CNFSM4JRO7DPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEM5A7ZI#issuecomment-591007717, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF2U5HRE3C7RWNHGBCLITLTREVRYNANCNFSM4JRO7DPA .

--

Dr. George Millward Cooperative Institute for Research in Environmental Sciences (CIRES) University of Colorado, Boulder

Visiting scientist at NOAA Space Weather Prediction Center Boulder, CO

george.millward@noaa.gov

[gmillward]

actually, I googled - easy to find

On Tue, Feb 25, 2020 at 12:29 PM George Millward - NOAA Affiliate < george.millward@noaa.gov> wrote:

Raffaele

Do you know the references/links for this ESMF bilinear regridding (and related stuff I guess) - would be very interested to see how this works.

Cheers

George.

On Tue, Feb 25, 2020 at 11:48 AM rmontuoro notifications@github.com wrote:

SWIO does not perform regridding, but relies on regridding carried out by standard NUOPC connectors to increase flexibility. Such regridding uses standard ESMF methods and is unrelated to the WAM->IPE regridding method implemented in the SWPC mediator.

The provided output files were generated by using default ESMF bilinear regridding. No cell average is performed in this method. I believe this may be the reason why features related to IPE's flux tubes appear in the output values. We can investigate other regridding methods next.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NOAA-SWPC/IPE/issues/7?email_source=notifications&email_token=AF2U5HRTB3OQAWMLHS4RINDREVRYNA5CNFSM4JRO7DPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEM5A7ZI#issuecomment-591007717, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF2U5HRE3C7RWNHGBCLITLTREVRYNANCNFSM4JRO7DPA .

--

Dr. George Millward Cooperative Institute for Research in Environmental Sciences (CIRES) University of Colorado, Boulder

Visiting scientist at NOAA Space Weather Prediction Center Boulder, CO

george.millward@noaa.gov

--

Dr. George Millward Cooperative Institute for Research in Environmental Sciences (CIRES) University of Colorado, Boulder

Visiting scientist at NOAA Space Weather Prediction Center Boulder, CO

george.millward@noaa.gov

[twfang]

Some comparisons of plasma density and TEC.

[twfang]

Here are some plots from the original HDF5 output on IPE native grid. X and Y axes are magnetic longitude and latitude.

[gmillward]

Interesting - results at 350 look very similar

...and when you look at 150km - the 'artifacts' in the New plot can also be seen in the old one - just not as bad in the old one

all in all - better than I was expecting

On Thu, Feb 27, 2020 at 9:56 AM twfang notifications@github.com wrote:

Here are some plots from the original HDF5 output on IPE native grid

[image: O+_150_h5] https://user-images.githubusercontent.com/22968399/75466361-4dbcf280-5947-11ea-81d5-777a3f06c25d.png

[image: O+_350_h5] https://user-images.githubusercontent.com/22968399/75466366-501f4c80-5947-11ea-8cbc-9ecfe9f5e97b.png

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NOAA-SWPC/IPE/issues/7?email_source=notifications&email_token=AF2U5HUTVBJO5E4TRIIQRTTRE7WDLA5CNFSM4JRO7DPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENFDDUA#issuecomment-592064976, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF2U5HTXGUBALSE3HBQLG2DRE7WDLANCNFSM4JRO7DPA .

--

Dr. George Millward Cooperative Institute for Research in Environmental Sciences (CIRES) University of Colorado, Boulder

Visiting scientist at NOAA Space Weather Prediction Center Boulder, CO

george.millward@noaa.gov

[twfang]

Here are some neutral fields from the original HDF5 output on IPE native grid. X and Y axes are magnetic longitude and latitude. These fields are all looking like what WAM provides.

[twfang]

Some comparisons from different heights

[gmillward]

definitely a problem at lower altitudes and seems to disappear by 300km - interesting

On Thu, Feb 27, 2020 at 4:04 PM twfang notifications@github.com wrote:

Some comparisons from different heights [image: io_comp_alt] https://user-images.githubusercontent.com/22968399/75495014-c7ba9f00-597a-11ea-9f80-ec1612dd77b1.png

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NOAA-SWPC/IPE/issues/7?email_source=notifications&email_token=AF2U5HWHKQPJGSIQMXPA6LTRFBBG3A5CNFSM4JRO7DPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENGJTNY#issuecomment-592222647, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF2U5HVG3NJDDUAQFVLQ64LRFBBG3ANCNFSM4JRO7DPA .

--

Dr. George Millward Cooperative Institute for Research in Environmental Sciences (CIRES) University of Colorado, Boulder

Visiting scientist at NOAA Space Weather Prediction Center Boulder, CO

george.millward@noaa.gov

[twfang]

Results of O+ from outputs of SWIO on IPE original grid.

Results look similar at 200 and 300 km, but at 120 km results look quite different. Look at 120km results, it looks very much like the one that Geroge's package provided.

[rmontuoro]

Plots of WAM neutral fields remapped to a two-dimensional, uniform lat/lon grid at each pressure level can now be created from SWIO output since WAM's original mesh was updated to fully represent the model's reduced Gaussian grid (see issue https://github.com/NOAA-SWPC/WAM/issues/17).

[twfang]

Here are some comparisons of new and old O+ from different altitudes.

OLD (90-185km)

NEW (90-185km)

OLD (190-285km)

NEW (190-285km)

[twfang]

Here are some new results from Raffaele.

Latest (90-185km) with Raffaele's rework

Latest (190-285km) with Raffaele's rework

The old one in this plot uses George's method.

The results look much better! Thanks @rmontuoro Two things that I am still concerned,

1. The white spots in TEC plot are NaN values. These NaNs came from all minor species (O2+, NO+, oplus2d, oplus2p).
2. All the ions only contain values up to 780km, we do need values higher up.

[twfang]

Drift from 3/16 00:57 UT. I think the results look good. We should probably try to get the SWIO into the latest branch.

[timfullerrowell]

Thanks Tzu-Wei, Excellent.

Are these geographic northward, eastward, and vertical? Is it EXB only, or does it include parallel drifts. If EXB only, shouldn't vertical drift go through at some point at low latitude, at the geomagnetic equator? Are these from Raffaele's SWIO?

Tim

On Mon, Mar 16, 2020 at 9:49 AM twfang notifications@github.com wrote:

Drift from 3/16 00:57 UT.

[image: dft] https://user-images.githubusercontent.com/22968399/76775661-4a10d480-676b-11ea-9455-cd77b450addf.png

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NOAA-SWPC/IPE/issues/7#issuecomment-599610827, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH5BFOB53L4RUC3Y5JS6Q5DRHZC6TANCNFSM4JRO7DPA .

[twfang]

These are from Raffaele's SWIO. Only ExB drifts are included. They are at the geographic frame not geomagnetic.

[rmontuoro]

The SWIO component addresses the requirements listed in this issue. It is introduced in PR #21.