potential issue in varying Mesh_height_max

[naomimaruyama]

Although PR#290 has been merged and closed, we would need to further investigate the possible cause of the strange TEC reported in PR#290 when setting Mesh_height_max=1002 [km] (the default value is 782km). Furthermore, the 24hr WAM-IPE test run could not be finished within the 8hr walltime limit because the code hung up.

The following 2 next steps were suggested during today's meeting: (1) check Ne (and other variables) at different heights as the bad values could be happening only in a limited height range. (2) try another test with an altitude below 1000km in case there is a hidden boundary at 1000km.

This issue has been assigned to @IonospherePlasmasphereElectrodynamics and @rmontuoro.

[naomimaruyama]

(1) check Ne (and other variables) at different heights as the bad values could be happening only in a limited height range.

Both Ne@300km and NmF2 seem to show the strange features that are consistent with TEC, while Tn@300km appears to have no problem.

(a)Ne@300km

(b)NmF2

(c) Tn@300km

[naomimaruyama]

(2) try another test with an altitude below 1000km in case there is a hidden boundary at 1000km.

A new test, in which mesh_height_max=782km—>902km (instead of 1002km) is under investigation.

[akubaryk]

NOAA-SWPC/WAM-IPE#211 also displayed latitudinal cutoffs in the IPE post-processed output. We may want to look at native grid output and check through NOAA-SWPC/WAM-IPE#211 to provide some insight into this issue.

[akubaryk]

We found in the fixed-height IPE files that the WAM fields go to zero above 800km -- for example, some quick plots of the neutral temperature from ncview /scratch4/NCEPDEV/stmp4/Naomi.Maruyama/PR290after1002/netcdf/IPE_State.201303160100.nc

795km

800km

805km (all zeroes up to the top)

This pattern is seen in other neutral values coming from WAM, and likely contributes to the FLIP solver breaking.

[oehmke]

Is this happening in the IPE grid or in the intermediate fixed height grid between WAM and IPE? (Sorry, I was initially thrown off a bit by the term "fixed-height" used in this discussion.) If the zeros are in the IPE grid after interpolation, then this makes sense to me. The intermediate fixed height grid between WAM and IPE only goes up to 802km. If you interpolate to an IPE grid higher than that then you'll get zeros above that height (unless you make the intermediate grid taller). I thought that the extrapolation upward from the top of the intermediate grid was handled in IPE. However, now I wonder if what's happening is that IPE is extrapolating upward from mesh_height_max, so that once mesh_height_max goes above the top of the intermediate grid, then there's a gap that gets filled with zeros which are then extrapolated upward by IPE.

[naomimaruyama]

Hi @oehmke

Thank you very much for your clarification. This happens in the neural variables in the fixed height grid that was created by the post processing routine transforming from the ipe native flux tube grid to the fixed height grid. This indicates that zero values in the neutral parameters in the ipe grid.

The intermediate fixed height grid between WAM and IPE only goes up to 802km

It looks like we totally misunderstood what Raffaele changed. We thought that the interpolation would be done up to the appropriate height as long as one assigns the mesh_height_max.

We will need to double check exactly how the extrapolation was changed (or not) by Raffaele in both ipe cap and inside ipe associated with mesh_height_max.

thanks Naomi

On Mon, Jun 24, 2019 at 1:56 PM oehmke notifications@github.com wrote:

Is this happening in the IPE grid or in the intermediate fixed height grid between WAM and IPE? (Sorry, I was initially thrown off a bit by the term "fixed-height" used in this discussion.) If the zeros are in the IPE grid after interpolation, then this makes sense to me. The intermediate fixed height grid between WAM and IPE only goes up to 802km. If you interpolate to an IPE grid higher than that then you'll get zeros above that height (unless you make the intermediate grid taller). I thought that the extrapolation upward from the top of the intermediate grid was handled in IPE. However, now I wonder if what's happening is that IPE is extrapolating upward from mesh_height_max, so that once mesh_height_max goes above the top of the intermediate grid, then there's a gap that gets filled with zeros which are then extrapolated upward by IPE.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/SWPC-IPE/WAM-IPE/issues/299?email_source=notifications&email_token=ADIZZYVQW635QGKA7RHJPK3P4ERGNA5CNFSM4HSW2PL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYOBDTI#issuecomment-505156045, or mute the thread https://github.com/notifications/unsubscribe-auth/ADIZZYWLUGVSX5KCPTM4PKDP4ERGNANCNFSM4HSW2PLQ .

--

Naomi Maruyama CIRES, Univ. of Colorado Boulder NOAA Space Weather Prediction Center phone: +1 303.497.4857 <%28303%29%20497-4857> email: naomi.maruyama@noaa.gov

-- Sent from Gmail Mobile

[twfang]

@IonospherePlasmasphereElectrodynamics Naomi, could you please let us know who needs to check the extrapolation and what exactly needs to be done? Thanks!

[naomimaruyama]

More eyes would help us find bugs quicker.

As for ipe cap, could Bob and Raffele both please double check the extrapolation?

as for inside ipe (neutral/module_neutral.f90), i am going to double check it as soon as I find the time. Also could either Tzu-Wei or George also please double check the extrapolation too? Thanks.

On Wed, Jul 3, 2019 at 6:23 PM twfang notifications@github.com wrote:

@IonospherePlasmasphereElectrodynamics https://github.com/IonospherePlasmasphereElectrodynamics Naomi, could you please let us know who needs to check the extrapolation and what exactly needs to be done? Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SWPC-IPE/WAM-IPE/issues/299?email_source=notifications&email_token=ADIZZYXQYBRPRP2YU74S5A3P5THBRA5CNFSM4HSW2PL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZE7G2Y#issuecomment-508162923, or mute the thread https://github.com/notifications/unsubscribe-auth/ADIZZYV7KMAHDWJIGNCCUPTP5THBRANCNFSM4HSW2PLQ .

--

Naomi Maruyama CIRES, Univ. of Colorado Boulder NOAA Space Weather Prediction Center phone: +1 303.497.4857 <%28303%29%20497-4857> email: naomi.maruyama@noaa.gov

[rmontuoro]

To make sure we promptly address any outstanding issues in the way of development and effectively prioritize our upcoming tasks, I'd like to summarize the current status of WAM-IPE's development code:

1. PR#290 introduced the ability to fill polar and equatorial regions in IPE's 3D coupling mesh, which is generated in IPE's NUOPC cap from the IPE magnetic grid in order to exchange fields with WAM. Originally empty polar and equatorial mesh regions are now filled by default. This can be reversed by setting mesh_fill = 0 in the &ipecap input namelist.

2. The maximum height (km) of IPE's coupling mesh can be set by input namelist variable mesh_height_max. If sw_neutral is 1, this value is also used in IPELIB to determine the starting level for the vertical extrapolation of WAM neutral fields and O, O2, and N2 densities (see subroutine neutral() in IPELIB/src/neutral/module_neutral.f90). This feature has been available for a while and has not been changed in PR#290. The default value of mesh_height_max is 782 km, set in IPELIB/src/main/module_input_parameters.f90.

3. In WAM-to-IPE 1-way coupling, WAM fields are interpolated/extrapolated in the SWPC mediator to an intermediate 3D mesh, built from a set of predefined vertical levels. WAM fields on the intermediate 3D mesh are then regridded to the IPE 3D coupling mesh in order to determine their values on IPE's native magnetic grid. The intermediate 3D mesh was introduced in the SWPC mediator for efficiency, since it minimizes the number of vertical interpolation/extrapolation operations and leverages ESMF's ability to perform fast regridding between 3D meshes by pre-computing once the set of regridding weights (route handle) for the fixed vertical levels. However, this setup implies that WAM fields can only be transferred to IPE up to the maximum height of the intermediate 3D mesh. Note that the mediator's intermediate 3D mesh is currently built from file wam3dgridnew_20160427.nc, which includes 188 vertical levels extending up to 800 km as shown below:

$ ncdump -v height wam3dgridnew_20160427.nc
netcdf wam3dgridnew_20160427 {
dimensions:
    lons = 192 ;
    lats = 94 ;
    nlevels = 188 ;
variables:
    double lons(lats, lons) ;
        lons:standard_name = "longitude" ;
        lons:units = "degrees_east" ;
        lons:_FillValue = -999. ;
    double lats(lats, lons) ;
        lats:standard_name = "latitude" ;
        lats:units = "degrees_north" ;
        lats:_FillValue = -999. ;
    double height(nlevels) ;
        height:standard_name = "elevation" ;
        height:units = "kilometers" ;
    int NumPerRow(lats) ;
        NumPerRow:standard_name = "Number of nodes per row" ;
    int ShuffleOrder(lats) ;
        ShuffleOrder:standard_name = "Shuffle Order for Distribution" ;
data:

 height = 0.34, 0.39, 0.44, 0.51, 0.58, 0.66, 0.75, 0.86, 0.97, 1.11, 1.26, 
    1.42, 1.61, 1.82, 2.05, 2.31, 2.59, 2.91, 3.25, 3.62, 4.03, 4.46, 4.93, 
    5.42, 5.95, 6.51, 7.09, 7.7, 8.33, 8.99, 9.66, 10.35, 11.04, 11.75, 
    12.47, 13.2, 13.93, 14.67, 15.42, 16.19, 16.97, 17.76, 18.58, 19.42, 
    20.27, 21.15, 22.05, 22.97, 23.92, 24.88, 25.87, 26.88, 27.91, 28.96, 
    30.04, 31.15, 32.29, 33.46, 34.66, 35.9, 37.18, 38.49, 39.85, 41.25, 
    42.68, 44.15, 45.65, 47.18, 48.74, 50.31, 51.9, 53.49, 55.09, 56.69, 
    58.29, 59.88, 61.46, 63.03, 64.58, 66.13, 67.66, 69.18, 70.69, 72.2, 
    73.72, 75.2, 76.69, 78.16, 79.63, 81.1, 82.57, 84.02, 85.48, 86.93, 
    88.37, 89.81, 91.25, 92.68, 94.11, 95.54, 96.97, 98.41, 99.86, 101.33, 
    102.83, 104.38, 105.99, 107.66, 109.44, 111.33, 113.36, 115.61, 118.11, 
    120.92, 124.09, 127.6, 131.47, 135.68, 140.25, 145.17, 150.44, 156.05, 
    161.98, 168.24, 174.8, 181.66, 188.81, 196.24, 203.93, 211.9, 220.13, 
    228.61, 237.34, 246.32, 255.54, 265, 274.69, 284.6, 294.72, 305.04, 
    315.56, 326.27, 337.22, 348.31, 359.6, 371.08, 382.77, 394.69, 406.85, 
    418, 430, 440, 450, 460, 470, 480, 490, 500, 510, 520, 530, 540, 550, 
    560, 570, 580, 590, 600, 610, 620, 630, 640, 650, 660, 670, 680, 690, 
    700, 710, 720, 730, 740, 750, 760, 770, 780, 790, 800 ;

In conclusion, mesh_heigh_max needs to be set consistently with the vertical levels used to build the mediator's intermediate 3D grid.

Also, since WAM neutral fields and densities can be vertically extrapolated in both the IPE and the SWPC mediator, the most efficient setup would use an intermediate 3D mesh extending up to WAM's highest vertical level.

[twfang]

The IPE levels are now sent to the SWPC mediator so there's no risk of inconsistency that was described in this issue. Also, the current outputs do not show the same problems in ionospheric parameters. Therefore, the issue is closed.