IPE Physics: IPE stability issue

[twfang]

Under certain WAM conditions, WAM-IPE coupled run produces strange feature in O+ density, velocity, and eventually molecular ions also become NaNs. To debug this issue, we will run the model with time-vary drivers (timeobs) with empirical E fields for the 2013 St. Patrick Day storm period using 20130316 WAM initial condition under /scratch3/NCEPDEV/swpc/noscrub/Adam.Kubaryk/ICS/WAM-IPE/2013031600

and IPE initial condition under /scratch3/NCEPDEV/swpc/noscrub/refactored_ipe_input_decks/IPE_State.apex.201303160000.h5

I will then report TEC, O+ density, O+ velocity from this run to provide information to debug. Everyone who is doing the same run on different systems feels free to report plots here.

[wanghj2015]

Re-did Adam's 'legacy IPE code (development)' run, with one fix in gsm/dyn/get_w_z.f (remove negative values in species):

         !rqg(i,lan,k) = grid_gr(i+jlonf,g_rq-1+k)
          rqg(i,lan,k) = max(grid_gr(i+jlonf,g_rq-1+k), qmin)

The 'yellow' panels disappeared, and 'NaN's don't appear in molecular ions in the 8 day run done:

[akubaryk]

Changing the compilation optimization for IPE to -O0 or -O2 resolved the NaN's running from 2013031600 initial conditions with timeobs input parameters. We performed a restart with both options from the same Phase 3 2013032000 forecast, right before the NaN's would normally appear running with -O3. @twfang will provide visuals.

[twfang]

George's fix mitigates the NaN issue that is in IPE. A 6-day run starting from March 16 doesn't show any NaNs in TEC toward the end of the run. However, the ionosphere starts to look odd into March 21, which requires more investigation!

This run uses the time-varying drivers and empirical electric fields. 2013, March 20

2013, March 21

[gmillward]

OK - I'll have a look at what is going on with those TEC values on the 21st

Cheers

George.

On Mon, Nov 4, 2019 at 1:35 PM twfang notifications@github.com wrote:

George's fix mitigates the NaN issue that is in IPE. A 6-day run starting from March 16 doesn't show any NaNs in TEC toward the end of the run. However, the ionosphere starts to look odd into March 21, which requires more investigation!

This run uses the time-varying drivers and empirical electric fields. 2013, March 20 [image: new_fix_0320] https://user-images.githubusercontent.com/22968399/68155613-e219ca00-fefe-11e9-85de-4284a3db62d2.png

2013, March 21 [image: new_fix_0321] https://user-images.githubusercontent.com/22968399/68155891-7e43d100-feff-11e9-9c7a-389ba048080c.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SWPC-IPE/WAM-IPE/issues/348?email_source=notifications&email_token=AF2U5HXGIMZLX6WTSHYIAV3QSCBPHA5CNFSM4IZRF3V2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDATZYY#issuecomment-549534947, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF2U5HTLISGGI6MU2LQSVBDQSCBPHANCNFSM4IZRF3VQ .

--

Dr. George Millward Cooperative Institute for Research in Environmental Sciences (CIRES) University of Colorado, Boulder

Visiting scientist at NOAA Space Weather Prediction Center Boulder, CO

george.millward@noaa.gov

[twfang]

Testing the new repository GSMWAM-IPE with new build system, results are consistent with previous WAM-IPE. However, looking into IPE outputs on 0320, the failed flux tubes don't occur at the same location but occur at the same UT time as before.

0320

[gmillward]

is this on Hera or WCOSS

Today I can give you the 1 character fix!!

On Wed, Nov 13, 2019 at 2:17 PM twfang notifications@github.com wrote:

Testing the new repository GSMWAM-IPE with new build system, results are consistent with previous WAM-IPE. However, looking into IPE outputs on 0320, the failed flux tubes don't occur at the same location but occur at the same UT time as before.

0320 [image: tec_0320] https://user-images.githubusercontent.com/22968399/68805065-51d73580-0620-11ea-8007-aa0dcfc9a5bb.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NOAA-SWPC/WAM-IPE/issues/348?email_source=notifications&email_token=AF2U5HVYEXI7R75CJDXXKSLQTRVILA5CNFSM4IZRF3V2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED7V43I#issuecomment-553606765, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF2U5HQXF2IWCLTI3EJN7YTQTRVILANCNFSM4IZRF3VQ .

--

Dr. George Millward Cooperative Institute for Research in Environmental Sciences (CIRES) University of Colorado, Boulder

Visiting scientist at NOAA Space Weather Prediction Center Boulder, CO

george.millward@noaa.gov

[twfang]

That was on Hera. I have tested the same run on WCOSS. TEC results look similar to what we had before, the Nans show up at the same time with slightly different locations. The new repository now has everything we need on WCOSS.

[twfang]

@IonospherePlasmasphereElectrodynamics
Please upload your old IPE 10 day run TEC in this issue. Thank you!

[twfang]

With George's new fixes, we are able to mitigate the NaN issue that we have been dealing with in the past few months. Here is the results of TEC after 5-day run starting from 0UT on March 16, 2013.

March 20, 2013

The plan is to merge the changes into the new repo and go from there!