Bias correction of TEC of operational WAM-IPE against GloTEC TEC

[ZhuxiaoLi66]

Bias correction will apply the TEC of operational WAM-IPE against GloTEC TEC. The related former work can be found in the old issues of https://github.com/NOAA-SWPC/WAM/issues/12 and https://github.com/NOAA-SWPC/GSMWAM-IPE/issues/55

[twfang]

@ZhuxiaoLi66 could you include more descriptions when you upload any plots? That will help to keep these records useful. Thank you!

[ZhuxiaoLi66]

Sure, it was just a quick plain result to discuss with Tim just now. Please have a look at the following plots. The daily TEC value for each day is the mean of hourly results.

[ZhuxiaoLi66]

The above plots are the bias ratio of the operational WAM-IPE against GloTEC, the following is the bias ratio of the free WAM-IPE run during the same period against GloTEC. It is for comparison and further test runs.

[ZhuxiaoLi66]

we can see the operational WAM-IPE results are close to free WAM-IPE, except the bias increase a little bit in the last few days.

[timfullerrowell]

Yes, looks very similar. So we can use the free run to set the Kt when we have a longer period.

Be nice to add the MIT and LISN in the 70W latitude slice plots when they become available to make sure Glo-TEC is consistent.

On Wed, Aug 25, 2021 at 12:58 PM ZhuxiaoLi @.***> wrote:

we can see the operational WAM-IPE results are close to free WAM-IPE, except the bias increase a little bit in the last few days.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NOAA-SWPC/GSMWAM-IPE/issues/54#issuecomment-905792546, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH5BFOCT7TAJWOD23ACC2W3T6U4L7ANCNFSM5CIJ4A3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[ZhuxiaoLi66]

Got it, thanks. yes, the availability of LISN and MIT is an issue for now.

[ZhuxiaoLi66]

The following plots are the WAM-IPE/GloTEC bias ratio for the operational runs and free runs respectively, for a longer period, 20210616-0802.

[ZhuxiaoLi66]

For estimating the TEC adjustment range by changing keddy0 (eddy mixing coefficient) for the 0616-0717 2021 case (see former results above), I conducted 6 free WAM-IPE runs during this period with different keddy0 (80, 100, 140, 180, 220, 260, 300) respectively. The corresponding bias correction ratios are shown as below:

[ZhuxiaoLi66]

The green line with skeddy0=140 is the baseline. The ratio of the TEC of each skeddy0 to baseline is

skeddy0 80 100 140 180 220 260 300 ratio 1.08 1.05 1. 0.96 0.92 0.89 0.86

With the plot, we can see the adjustment range of TEC based on Keddy0 is not very fast, to get a 20% change, it takes the skeddy0 change from 260 to 80.

also it is limited at the increase end, with the eddy mixing code below keddy(:) = skeddy0 + skeddy_semiann(cos(4.pi*(dayno+9.)/365.)) given skeddy_semiann=60., the skeddy0 should be above 60. to get keddy>0 for physics meaningful.

[ZhuxiaoLi66]

The following plots indicate the results of our experiments on reducing TEC by reducing the drift velocity. Our methodology are changing the value of v_boost factor in subroutine of Calculate_ExB_Velocity in IPE/src/ipelib/IPE_Electrodynamics_Class.F90. We reset the value of v_boost factor into 0.5, 0.6, 0.8 respectively instead of the original value of 1.0. The tests are done for both the case of Keddy0= 140 (default) and Keddy0=80. The results are showing as following, the third plot are the results of our keddy0 experiments as a comparison.

[ZhuxiaoLi66]

We can see the change of the v_boost_factor from 1.0 to 0.5 can cause the decrease of the TEC bias ratio to about 0.1, reduce TEC by about 10%.
@twfang @timfullerrowell my question is that does the increasing of v_boost_factor (to above 1.0) to increase the drift velocity have physical significance? In our July case, we would like to figure out some way to increase the TEC. By the way, several WAM-IPE runs for a longer period (7/18-9/20) are running for checking the trends until recent time.

[ZhuxiaoLi66]

I also plan to make some plots on 70W and 20UT to see the effect of the reduction of the TEC in WAM-IPE makes it closer to GloTEC and other observational data. This is respected based on Tzu-Wei's results showed last Monday,

[twfang]

@ZhuxiaoLi66 Do you have a map that shows where are the TEC locations you use in your bias correction? Currently, our WAM-IPE drift appears to be too large, there is probably no need to increase the v_boost_factor.

[ZhuxiaoLi66]

@twfang Got it, thanks. You are correct, I also recalled the peak area of TEC of WAM-IPE arround S20 and N20 are higher than observations (MIT TEC). At least along 70W and arround the peak area.

[ZhuxiaoLi66]

The following is a typical TEC map of MIT TEC.

[ZhuxiaoLi66]

The following plots are the TEC of GloTEC and free WAM-IPE along 70W at 20UT during the period of 20210616-0717, the two cases are v_boost=1.0 and v_boost=0.8 respectively. We can see the decrease in the v_boost coefficient reduces the drift velocity to some degree, especially around the peak area of S20 and N20 along 70W, more tests will be done soon.

[ZhuxiaoLi66]

The bias ratio of WAM-IPE/GloTEC seems to hold for a longer period after Jul.18, the plots will be shown here soon.

[ZhuxiaoLi66]

[ZhuxiaoLi66]

@timfullerrowell @twfang The above plot shows a longer period for the bias ratio of WAM-IPE/GloTEC. the whole time series until Oct. 15 will show after the data copying to WCOSS Venus by the WCOSS help team, hopefully soon. In addition, I have downloaded the MIT TEC data from the MIT website for the whole period (20210618-1015), will do the bias ratio and the comparison in short time. As for the LISN data, I checked several times following the video Cesar gave and don't think the eqTEC data during 20210618-1015 is available on that website now, will confirm with Cesar again (sent an email already). Maybe need Tzu-Wei's help on this.

[ZhuxiaoLi66]

The following plot shows the TEC bias ratio of WAM-IPE/GloTEC with the 6 different Keddy0 during the whole period of 20210618-20211010.

[ZhuxiaoLi66]

The work for making a plan for operational bias correction is going on. So as several other works. The plan in detail will come out this Friday.

[ZhuxiaoLi66]

[ZhuxiaoLi66]

The plan for the Operational Bias Correction project has been made and is Automation of TEC bias correction plan new.docx attached here. Any comments and input are welcome.

[ZhuxiaoLi66]

[ZhuxiaoLi66]

The above plot is the bias ratio of WAM-IPE/MIT data for the period of 20210618-20210631, as a test for the whole period comparison of 20210618-20211010.

[ZhuxiaoLi66]

With George's great help, now the cron job (at 11UT each day) on bes-r2o-lx can utilize the python code to enter the operational WAM-IPE and GloTEC data on the same machine (/data) to calculate the hourly bias ratio (M/O) and write the daily value into a text file for the date of current_date-2, all the work can do automatically.

[ZhuxiaoLi66]

Please have a look at our new plan for this project, we are on the online track now. Automation of TEC bias correction plan new2.docx

[ZhuxiaoLi66]

the running mean calculation & plotting work is going on.

The following plot is a daily bias trend for recent days (Jan20-Feb5).

[ZhuxiaoLi66]

All, please have a look at the following plot, it is an old one for the period of Jan.5 - Jan.29. You can find the drop-down start from the 6th day (6th+5=11th!) and bounce up from (23th+5 = 28th) date as we saw in the former plot I showed at the meeting (above).

[ZhuxiaoLi66]

Today's daily bias correction figure.

[ZhuxiaoLi66]

Now we need an 'interval webpage solution' to upload the figure to a webpage or google drive directly from bes-r2o-lx. on bes-r2o-lx, I tried the following command to upload the plot to my google drive, which has failed.

curl today-plot https://drive.google.com/mydirectory

[akubaryk]

Uploading directly to Drive from the command line is an awful process that you shouldn't go through unless absolutely necessary. It's may be more convenient to simply email it to yourself as long as the attachment isn't too large.

echo "" | mail -s "subject" -a file_to_attach email@address.ext

[ZhuxiaoLi66]

Adam, Thanks for the very helpful comments. going to give it a try now.

[ZhuxiaoLi66]

This command can be in python code? each "" ? what is supposed in the quote? email@address.ext = zhuxiao.li@noaa.gov ?

[akubaryk]

You'd run that from the command line and substitute your own values in for subject, the file to attach, and your e-mail address.

[ZhuxiaoLi66]

got it, thanks.

[ZhuxiaoLi66]

Adam, I got the plot from r2o machine in my email box, thanks!

[ZhuxiaoLi66]

can we do multiple email addresses? or maybe a group email address later.

[akubaryk]

You can try comma separating the email addresses, e.g. zhuxiao.li@noaa.gov,tim.fuller-rowell@noaa.gov.

[ZhuxiaoLi66]

Thanks, will give a try later,