3D model instability

[williameaton]

Hi Kuangdai,

I hope you are well. I have three 3D models which keep producing instabilities when trying to run. They generally start to blow up ~ 30-50 % of the way through the simulation. I have attempted lowering the Courant number, as well as altering the dominant frequency of the mesh from 2 Hz to 2.1 Hz and 2.3141592 Hz in the hope that this would offset any sharp velocity boundaries from the boundaries of the mesh elements. Unfortunately neither of these worked. Finally, I previously attempted tapering the edges of the perturbed regions so that the boundaries were not as sharp, but this also did not solve the issue. I was wondering whether you had any suggestions for alterations to the mesh/model/inparam files that may make these simulations stable.

I attach below a zip file containing the input files and 3D models. inputs.zip

It is worth noting that similar 3D models (eg. with spheres of radius 1.5 and separation 3, instead of radius 1 and separation 2) run fine.

Any advice on this would be amazing!

Best wishes,

Will

[kuangdai]

Can you try halve the courant number? If it is not working, try to decrease the perturbation from 20% to 5%. In the file, I only see 20% and 0. If you have done boundary smoothening, there should be other values between 20% and 0.

[williameaton]

The Courant number is already at 0.15, but i'll give it a go! Yes, these models didnt have smoothing but having run other simulations that had instabilities, the smoothing didnt make any difference. I will try re-running again with smoothing and with a courant of 0.075.

[kuangdai]

You can try a bigger Nr as well. 360 may be too sparse so it caused overshooting in the azimuthal dimension.

[williameaton]

Thanks, I'll give that a go.

[tnissen]

And this would cause numerical instabilities ? It just seems bizarre given how the other simulations very nearby in parameter space worked.

On 31/03/2021 13:58, Kuangdai Leng wrote:

You can try a bigger Nr as well. 360 may be too sparse so it caused overshooting in the azimuthal dimension.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kuangdai/AxiSEM-3D/issues/17#issuecomment-811292104, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUMH5ELTTRBIBCKF74D4OTTGNPDNANCNFSM42EBVVYA.

-- <>--<>--<>--<>--<>--<>--<>--<>--<>--<>--<>--<> Associate Professor of Geophysics Dept. of Earth Sciences, Oxford University South Parks Road, Oxford OX1 3AN; UK www.earth.ox.ac.uk/people/tarje-nissen-meyer <>--<>--<>--<>--<>--<>--<>--<>--<>--<>--<>--<>

pronouns: he/him/his

[kuangdai]

Sparse sampling on the azimuthal direction may cause overshooting. As you can imagine, a few nearby points just sampled the peak and valley values, and of course this is model-dependent and sort of random triggering.

The simulation blows up in the middle instead of at the beginning. This means the problem localised in a small region.

[williameaton]

Hi Kuangdai,

Just to update you. I tried re-running with nr of 800 instead of 360 and instability still grows early on ~ 40 % of the way through simulation.

[kuangdai]

This input is too large for local inspection. Can you reproduce the error with a smaller mesh and Nr so that it can be run on one processor. I cannot debug under MPI, which will take ages.

[williameaton]

Sorry for the slow reply. I tried to re-install axisem locally (new computer) to run these smaller simulations but I'm getting an issue with the compiler when I try to make axisem (command line output is attached in a text file - I think its an issue with finding mpicxx but Im not very confident on this stuff).

I have attached a zip with the following files:

• Normal inparam files I use
• Install issue command line output + copy of build directory containing CMake logs etc. (if you want to take a look but no worries I know youre super busy!)
• Smaller homo2Hz.e mesh (15 km radius instead of 125 km)
• Smaller corresponding 3D model, called model.nc, as well as the model-generating script (variables in the bottom ~ 10 lines can be changed to produce different models/change resolution etc)
• Currently nr is set to 5 instead of 360
• loop_info_interval is set to 10 as I was aiming to check how long it would take on one processor.

I cant run it locally and produce the error until I can sort out the axisem install - If you are able to run it on your end that would be amazing. I anticipate (hope) it will blow up as usual, but cant be sure until its run.

In the mean time I will try and get axisem downloaded (sorry!) mini_domain.zip

[williameaton]

Hi Kuangdai, I just ran the aforementioned smaller simulation on archer to circumvent the installation issues. Seems to run fine in this case. I guess there must be some issue when increasing the nr/size of the simulation? I'll try running some slightly larger ones to pinpoint when it blows up

[williameaton]

Further update - I ran with this same smaller model with nr = 100. Took ~ 8 minutes on one node on archer so hopefully managable on your local station if required. Record sections of Greens functions and 2 Hz BP Greens functions (below) show instability growing. Note that the amplitude of each trace is normalised hence why, because of the growing instability at the earlier stations, you cant see the first ~ 200 seconds of the GF for the first few stations.

Attached below are the logs from the develop section of the output. Please let me know if you require any other files. Thanks again for all your help so far!

loop_timers.log measured_element_costs.log preloop_diagnosis.log

[kuangdai]

The uploaded mini_domain.zip contains something incorrect. Without changing anything, it will stop with an error of wrong source depth. After using a correct source depth, it will cost 25 hours on my Mac with nr=100. It is impossible that one node on Archer (or any HPC on Earth) can be 25*60/8 times faster than an i7 CPU on Mac. Please verify.

[williameaton]

Yes, sorry I forgot to change the source depth. Possibly worth clarifying that each node on archer2 has 128 cores where as I assume the Mac has ~ 4-8? Do you need it to run on a single core for debugging? I will try and find the minimum nr at which I observe an instability.

[williameaton]

Have run simulations at the following nr values:

• 20
• 25
• 30
• 35
• 50
• 100

Using nr = 25 the simulation doesnt blow up, but it does for nr = 30. Hopefully nr=30 should be managable on Mac? I also note that currently the courant number is set to 0.25 for all of these simulations so I guess that could be increased to shorten run times?

[williameaton]

Hi Kuangdai - I was just wondering whether there was any update on this?

[kuangdai]

I am afraid not. Does it run with 20% perturbation instead of 40%?

[williameaton]

Sorry for the slow reply - I can't get access to my linux system at the moment so can't check this but the perturbations should be 20 %. It may show as 40 % in paraview because the spheres almost overlap, but I believe the max. value of the arrays in the netcdf file should be 20 % (I hope!)). My deadline for this project is coming up in the next week or so, so I think I will just have to do without these simulations. Thanks for your help in trying to solve the issue, I really appreciate you taking the time.

[kuangdai]

Are you sure about 20%? I have been testing this using the attached model file. The perturbation is -20% or 0% on most of the points but with some "isolated" points with -40% perturbation at the corners. Not sure how you get this model, but I assume this is what you want. These isolated points tend to cause overshooting by SEM.

model.nc.zip

[williameaton]

Hmm...I think I know what the issue will be in the model-generating script. I will have a look at that asap and check - it should be very easy to rectify and then i'll try re-running the simulations! Thanks!