adjtomo / seisflows

An automated workflow tool for full waveform inversion and adjoint tomography
http://seisflows.readthedocs.org
BSD 2-Clause "Simplified" License
172 stars 122 forks source link

Program received signal SIGILL: Illegal instruction - Acoustic/elastic problem #216

Open raulleoncz opened 1 month ago

raulleoncz commented 1 month ago

Hi @bch0w, I apologize for any inconvenience this may cause you. I have been trying to do some inversion examples using the acoustic, acoustic/elastic and elastic configurations and this what I have found:

  1. When running a purely acoustic problem I have seen that the misfit is not being calculated properly. The residual I got is zero and because of that, there is no information from the sensitivity kernels. Here I add an example of this:

image

  1. The misfit is calculated properly just when the model has S wave information (acoustic/elastic and purely elastic). Therefore, even if I want to do an acoustic inversion I have to define the S-wave velocity model and here is where I found my last issue which was:

Program received signal SIGILL: Illegal instruction. Backtrace for this error: Could not print backtrace: executable file is not an executable Could not print backtrace: executable file is not an executable

0 0x100b32f33

https://github.com/SPECFEM/specfem2d/issues/1 0x100b321db https://github.com/SPECFEM/specfem2d/pull/2 0x19990b583 zsh: illegal hardware instruction ./bin/xspecfem2D

Here my first thought in that moment was that the simulation was unstable because of the CFL, source frequency or grid dispersion but after checking a little bit I found that after 4000 time steps the 'Max absolute value of scalar field in fluid (acoustic)' and the 'Color image maximum amplitude' quickly skyrocketed to very large values:

Time step number 5 t = 0.0032 s out of 10000

We have done 4.00040001E-02 % of the total Max norm of vector field in solid (elastic) = 0.00000000 Max absolute value of scalar field in fluid (acoustic) = 1.53421927E-02 Color image maximum amplitude = 4.0870062075555325E-003

Time step number 2000 t = 1.5992 s out of 10000

We have done 19.9919987 % of the total Max norm of vector field in solid (elastic) = 1.48898596E-03 Max absolute value of scalar field in fluid (acoustic) = 236.648941 Color image maximum amplitude = 1.4626149786636233E-003

Time step number 4000 t = 3.1992 s out of 10000

We have done 39.9939995 % of the total Max norm of vector field in solid (elastic) = 1718.72803 Max absolute value of scalar field in fluid (acoustic) = 19952228.0 Color image maximum amplitude = 2.1745527217967010E+022

I remember you told me this was an issue related to specfem2d so I took the .bin files from the last updated model and I ran some simulations. What I found is that this problem occurs when we have S-wave velocity information. When we have a purely acoustic configuration, specfem2d has no problem to simulate the propagation. I already posted this as an issue in the specfem2d GitHub page for more information. These are some pictures of the wave propagation until the simulation got really unstable:

image image

Based on that, I have some doubts.

Finally, another thing I was thinking is to implement a min and max velocity limits to prevent the algorithm from selecting very low wave velocities during the optimization process, mostly in highly complex cases. I would like to try to implement this idea, could you give me some advice?

Thank you so much for all your help, I really appreciate it!

bch0w commented 1 month ago

Hi @raulleoncz, thanks for opening this issue.

I have not done any acoustic or acoustic+elastic simulations before, sorry. But skyrocketing amplitudes in a simulation usually suggests numerical instability to me, and the need for a smaller time step. The velocities assigned by the acoustic domains are perhaps to low to be stable with the given timestep? But again this is more of a SPECFEM issue than SeisFlows

Hard to tell, what are you using for 'data' in this example? Your waveform figure suggests that the obs and syn waveforms are exactly the same, leading to zero residuals.