V&V on Ops. WAM-IPE Aug. 2020

[ZhuxiaoLi66]

The Ops. WAM-IPE storm validations will be conducted to verify the output of 3D neutral (might include plasma later) fields (U, V, W, T, O, O2, N2, mass density, H, He) and 2D neutral& plasma fields ( TEC, O/N2) from the tip operational WAM-IPE, against the corresponding observational dataset, and also the corresponding former runs of WAM-IPE. The storm case will be the St. Patrick storm of 2015 (Mar.16-21 2015). Will includes (not limited to) the following validations, a) The vertical integral O/N2 (at pressure levels) against the track-by-track 24hrs GUVI data during Mar.16-21 2015, against the corresponding former runs. b) The thermosphere neutral total mass density (at about 400km) against point-to-point along with the satellite trail data of GRACE. c) TEC against the output & data of GloTEC & MIT. d) Tides analysis based on the hourly output of an annual run of WAM. ...

[twfang]

Lastest run with WAM-IPE v1.0 in 3 min and 5 min timesteps can be found in the following link. https://drive.google.com/drive/u/1/folders/1qttFQybCAA9joRIM_Or2RHlblTKc3f-a

[twfang]

Some preliminary ionospheric validation by Tim can be found here. https://drive.google.com/file/d/1GagCjj0bMhBvXlJ-mWGFCGcBRL4Xv6y6/view

[ZhuxiaoLi66]

The WAM-IPE 2015 St. Patrick's (Mar.16-Mar.21) storm run by the latest operational version has been conducted with the IC files from the latest updated-WAM-physics runs. The output cadence is 3mins for both WAM and IPE in stmp4, hourly in stmp2. For some unknown reason (maybe need further investigation by Raffaele), we only have gotten the 3mins output from both WAM & IPE for the last restart run (3runs in line for 6 days), which means we have 3mins output for 20150320 & 20150321 for now in the following directory, Hera:/scratch1/NCEPDEV/stmp4/Zhuxiao.Li/coupled_20150316_tip_validation_3mins. The hourly output of WAM & IPE for the total 6 days (20150316-20150321) are on the directory below: /scratch1/NCEPDEV/stmp2/Zhuxiao.Li/coupled_20150316_tip_validation_3mins

I already submitted a coupling simulation with 6 days in one run for the full 3mins output dataset with the latest operational version. While the output issue for the restart runs still need to be figured out for future application.

[ZhuxiaoLi66]

The following plot is the O/N2 validation for the hourly output of the current storm run on 20150317(most intense storm time), against the GUVI tracks data. That means the corresponding WAM panel in the following plot is a track-by-track plot on 20150317 with hourly WAM output.

[ZhuxiaoLi66]

The following is the corresponding validation for WAM 3mins NetCDF output in a WAM former run back to Dec. 2019. It is expected that the 3mins output from SWIO on 20150317 by the current operation should be very close to this result, since the physics & compilers. etc. haven't been changed since then.

[ZhuxiaoLi66]

This plot is a similar one with the plot above, except based on the 3mins output from SWIO (instead of the WAM direct netcdf-output for the above plot) of the tip operational GSMWAM-IPE. The WAM direct output represent the original output from WAM. After comparison, we can found that the results are close, while still different in some fine features.

[ZhuxiaoLi66]

Results of b) and c) will come out soon.

[ZhuxiaoLi66]

The SWIO output (for WAM) of the thermosphere total mean mass at 400km (iss_mass_density) for 2015 St. Patrick storm has been applied for the total mean mass validation of WAM against the corresponding GRACE orbit-by-orbit observational data. The results are shown in the following plots. Unscaled GRACE 2015 March 16_19 400km_den_iss_lonlat_2.docx

[ZhuxiaoLi66]

The bias of the WAM output from GRACE data at the intense storm time (12UT Mar.17 - 6UT Mar.18) is larger than the one in the former validation with the interpolated multiple fixed height WAM output as shown in the following plots.

[ZhuxiaoLi66]

The difference is considered due to the varying height (not fixed height) along the orbit of GRACE. Based on Martin's statement, minimum height: 392.29km, maximum height: 431.74km. The plan now is that the total mean mass dataset will be remade with the original SWIO output of thermosphere_mass_density, the interpolated fixed height data with height 380km-440km will be applied to do the validation again.

[ZhuxiaoLi66]

The dataset of total mean mass with a vertical dimension of fixed height has been remade based on the original SWIO output of thermosphere_mass_density. The new validation results (show below) against the along-orbit GRACE data looks very consistent with our former base validation results. it verified that the SWIO output in the WAM-IPE operational version keeps the consistent physics feature in neutral total mass during storm time with our base version.

[ZhuxiaoLi66]

Attached is the original word file for the latest validation. Unscaled GRACE 2015 March 16_19 SWIO_opr_valid_7levs.docx

[ZhuxiaoLi66]

[ZhuxiaoLi66]

The above plot is for the entire day of Mar.6 of 2013. The following plots are the on averaging for 20mins range on 20130306 00:20:00-00:40:00 and the corresponding one downloaded from CEDAR website. The difference due to the latter one is rebined into 3deg by 3 deg and the area calculation is weighted by cosin(lat).

[ZhuxiaoLi66]

[ZhuxiaoLi66]

[ZhuxiaoLi66]

With the MIT GPS TEC observational data, we are trying to estimate the WAM TEC bias based on the following formula.

bias = exp ( (SUM over N values of orbit timeseries ( ln (TEC(model) / TEC(obs) ) ) ) / N )

[timfullerrowell]

The first step is to develop the methodology using the observed (MIT maps) and modeled TEC for the period from March 6-15th. We will be using the expression for TEC bias as the metric:

bias = exp ( SUM over N values of ( ln (TEC(model) / TEC(obs) ) ) / N )

1. Firstly, choose one UT time interval of observed TEC from the MIT maps, with N observations at a set of lat/long locations
2. Extract the TEC from the set of WAM-IPE simulations with varying eddy mixing of composition at the same time
3. Interpolate WAM-IPE TEC to the same set of N values at the same lat/long as the observations (maybe averaged over the data time interval)
4. Calculate the bias using above expression and find the WAM-IPE map with the minimum bias
5. Repeat for all UTs for a typical day, and over the 10 day period to see if the results are consistent (the data values will cover different regions during the UT day so the results might not be consistent)

If the results are not consistent, try dividing up the regions a) day/night separately, b) low latitude to target EIA peaks in TEC, c) regions around the peaks +/-10degrees N/ and E/W, etc. until a reliable consistent is established

Repeat with Glo-TEC.

Once a reliable methodology has been developed. Set up a system to routinely track TEC bias on a weekly basis between WAM-IPE and Glo-TEC. For each week, determine if there is an increasing trend in the TEC bias and determine if an adjustment in the eddy mixing in the operational version is required.

[ZhuxiaoLi66]

[ZhuxiaoLi66]

[ZhuxiaoLi66]

Tim, the above is the TEC comparison at 12UT between WAM and MIT GPS data. Thanks.

[timfullerrowell]

Something doesn't look right. Is the MIT TEC plat a particular day, or an average from 6-15th?

On Tue, Jan 12, 2021 at 11:56 AM ZhuxiaoLi notifications@github.com wrote:

Tim, the above is the TEC comparison at 12UT between WAM and MIT GPS data. Thanks.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NOAA-SWPC/GSMWAM-IPE/issues/41#issuecomment-758866506, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH5BFOC7IBSUUIDBG4RPIWTSZSLNJANCNFSM4PTUASWQ .

[ZhuxiaoLi66]

the MIT TEC is an average for Mar.6-15th.

[timfullerrowell]

Do you have access to the MIT plots on particular days for that period, to make sure we have interpreted the format correctly (e.g., longitude vs LT). I just tried to go to the MIT website and it said there was a Python problem. Are you able to get in.

On Tue, Jan 12, 2021 at 2:51 PM ZhuxiaoLi notifications@github.com wrote:

the MIT TEC is an average for Mar.6-15th.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NOAA-SWPC/GSMWAM-IPE/issues/41#issuecomment-759057273, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH5BFOHL66ANOFASY3GWWSLSZS753ANCNFSM4PTUASWQ .

[ZhuxiaoLi66]

I also can't access the data plots (stuck in the middle of somewhere). I Will give a retry tomorrow or just send an email to Bill Rideout to report this tonight.

[ZhuxiaoLi66]

The time average (Mar.6-15 2013) MIT TEC becomes more consistent with the WAM output after I interpolated the data onto the IPE output grid (more reasonable grid). the following is the improved MIT TEC at 12 UT.

[ZhuxiaoLi66]

The WAM TEC output biases at specific UT with different eddy0 values have been calculated based on the following formula: The output of 20130306 was calculated as a test, the results are shown in the following plot.

[ZhuxiaoLi66]

the TEC bias averaged during Mar.6-15 2013 has been calculated. The pattern is very consistent with the pattern of day1 (Mar.6) and each of the other days. The following plots are the mean TEC bias and the TEC bias for several other days.

[ZhuxiaoLi66]

[ZhuxiaoLi66]

The following is the average MIT TEC & coverage at different UT.

[ZhuxiaoLi66]

[ZhuxiaoLi66]

The following is the MIT TEC interpolated onto WAM-IPE output grid for the bias calculation.

[ZhuxiaoLi66]

The following is the GloTEC data example from the latest 20210103-0120 run. The data is with quality_flag>=3, good.

[ZhuxiaoLi66]

The NCL code to calculate the model bias (WAM/GloTEC) is ready, the following is the code test to compare the GloTEC data (20210103-20210112) and WAM data (20130306-0315). Please notice that the time periods of the two data do not match, so it is just a code test.

[timfullerrowell]

Thanks Zhuxiao. Interestingly the UT trend is similar to MIT. Are the MIT maps for the January period also available? Tim

On Mon, Feb 1, 2021 at 2:38 PM ZhuxiaoLi notifications@github.com wrote:

The NCL code to calculate the model bias (WAM/GloTEC) is ready, the following is the code test to compare the GloTEC data (20210103-20210112) and WAM data (20130306-0315). Please notice that the time periods of the two data do not match, so it is just a code test.

[image: bias_M_O_TEC_WAM_GloTEC_test] https://user-images.githubusercontent.com/22546571/106521148-087c3b80-649b-11eb-99fc-c900d2b8f49b.png

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NOAA-SWPC/GSMWAM-IPE/issues/41#issuecomment-771176628, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH5BFOATHGN5JDDO5Q3M5M3S44NMVANCNFSM4PTUASWQ .

[ZhuxiaoLi66]

The following is the TEC maps example of GloTEC & MIT in January of 2021.

[ZhuxiaoLi66]

The GloTEC TEC during this period is much larger compared with MIT TEC, at a factor close to 2, the reason is investigating. The following plots show the TEC ratio (MIT/GloTEC) at different UT by two different interpolation ways on the observational data.

[ZhuxiaoLi66]

After excluding the area with TEC <10 in the bias calculation of MIT/GloTEC and WAM/MIT. We can see the bias between MIT&GloTEC is improved to close to 1. and the WAM/MIT is improved by becoming more converging in the diurnal variation.

[ZhuxiaoLi66]

With the WAM-IPE output from pseudo-operational (free) runs during the period of Jan.3-Jan.20, 2021, the WAM-IPE biases of TEC against GloTEC and MIT GPS data respectively for that period have been investigated. The results are shown below.

[ZhuxiaoLi66]

The difference of TEC between GloTEC & MIT is under investigation.

[ZhuxiaoLi66]

[ZhuxiaoLi66]