receiver preparation is too time consumming in xspecfem3d solver

[specfem3d-zhang-ksu]

In preparing 2562 receivers, Elapsed time for receiver detection is 3293 s. I am wondering if anybody has better idea to improve it. Also, if there is doubling in the mesh, and if the receivers are along a vertical line (VSP), the locations of the receivers near the doubling mesh can be overlap, meaning these receivers are at the same locations, even physically not.

[komatits]

In SPECFEM3D_GLOBE there is this flag in setup/constants.h.in to make source detection much faster:

! flag to exclude elements that are too far from target in source detection logical, parameter :: USE_DISTANCE_CRITERION = .true.

which you can easily adapt for receiver detection as well (it may already be, I do not remember; if so, please update the comment, changing "source detection" to "source and receiver detection") and then cut and paste it in SPECFEM3D (Cartesian).

That would be very useful. If you do it please create a Git pull request to merge that into the official source code.

In setup/constants.h.in of SPECFEM3D (Cartesian) there is also this:

! no Lagrange interpolation for seismograms (we then take the value at the closest NGLL point, which is less accurate but faster) logical, parameter :: FASTER_RECEIVERS_POINTS_ONLY = .false.

but then you get an approximate location instead of exact location for the receiver (you get the closest mesh point), which makes the seismograms a bit less precise.

Dimitri.

On 03/19/2015 02:55 PM, specfem3d-zhang-ksu wrote:

In preparing 2562 receivers, Elapsed time for receiver detection is 3293 s. I am wondering if anybody has better idea to improve it. Also, if there is doubling in the mesh, and if the receivers are along a vertical line (VSP), the locations of the receivers near the doubling mesh can be overlap, meaning these receivers are at the same locations, even physically not.

— Reply to this email directly or view it on GitHub https://github.com/geodynamics/specfem3d/issues/465.

Dimitri Komatitsch CNRS Research Director (DR CNRS), Laboratory of Mechanics and Acoustics, UPR 7051, Marseille, France http://komatitsch.free.fr

[specfem3d-zhang-ksu]

I see. Unfortunately, I can not commit my changes here. This is very inconvenient to me.

[specfem3d-zhang-ksu]

Thanks, Dimitri.

[mpbl]

Hello @specfem3d-zhang-ksu If you want to contribute your changes, there is a handful of instructions right there: https://github.com/geodynamics/specfem3d/wiki

Thanks,

[komatits]

Hi,

To commit changes, just follow this Wiki: https://github.com/geodynamics/specfem3d/wiki/Using-Git-for-SPECFEM

On 03/19/2015 03:20 PM, specfem3d-zhang-ksu wrote:

I see. Unfortunately, I can not commit my changes here. This is very inconvenient to me.

— Reply to this email directly or view it on GitHub https://github.com/geodynamics/specfem3d/issues/465#issuecomment-83600161.

Dimitri Komatitsch CNRS Research Director (DR CNRS), Laboratory of Mechanics and Acoustics, UPR 7051, Marseille, France http://komatitsch.free.fr

[specfem3d-zhang-ksu]

Dimitri,

I did follow that, but I always get an an error, which I sent you before. When I try to commit my change, I get an error $git push

error: The requested URL returned error: 403 Forbidden while accessing https://github.com/geodynamics/specfem3d.git/info/refs

fatal: HTTP request failed

[komatits]

This comes from your machine or your firewall; please contact your system administrator. (other users do not get this error message)

On 03/19/2015 03:32 PM, specfem3d-zhang-ksu wrote:

Dimitri,

I did follow that, but I always get an an error, which I sent you before. When I try to commit my change, I get an error $git push

error: The requested URL returned error: 403 Forbidden while accessing https://github.com/geodynamics/specfem3d.git/info/refs

fatal: HTTP request failed

— Reply to this email directly or view it on GitHub https://github.com/geodynamics/specfem3d/issues/465#issuecomment-83609520.

Dimitri Komatitsch CNRS Research Director (DR CNRS), Laboratory of Mechanics and Acoustics, UPR 7051, Marseille, France http://komatitsch.free.fr

[mpbl]

@specfem3d-zhang-ksu @komatits I think you have forgotten to fork the main repo.

Have you been through this step:

create a fork (a copy of the repository on your GitHub account) (do this only once, i.e. if you are using several computers, each with a copy of the code, then run "git fork" on the first machine only, no need to do it when you create copies on the other machines later)

$ git fork

[komatits]

yes, please follow the Wiki exactly (all steps are needed)

On 03/19/2015 03:40 PM, Matthieu Lefebvre wrote:

@specfem3d-zhang-ksu https://github.com/specfem3d-zhang-ksu @komatits https://github.com/komatits I think you have forgotten to fork the main repo.

Have you been through this step:

create a fork (a copy of the repository on your GitHub account) (do this only once, i.e. if you are using several computers, each with a copy of the code, then run "git fork" on the first machine only, no need to do it when you create copies on the other machines later)

$ git fork

— Reply to this email directly or view it on GitHub https://github.com/geodynamics/specfem3d/issues/465#issuecomment-83612033.

Dimitri Komatitsch CNRS Research Director (DR CNRS), Laboratory of Mechanics and Acoustics, UPR 7051, Marseille, France http://komatitsch.free.fr

[EtienneBachmann]

I totally agree with the fact that source and receiver location are the most time consuming stages before the timerun step. I try to turn on the flags that should accelerate this but I just won 1%. When looking at the source code, we can see the greedy time part (about 95%) : for each source (receiver), we loop on all the gll points of the MPI slice to check the distance to the source(receiver). There is probably an other way to do that, maybe with a Kd-Tree or something like this. I think this problem worths working on it, because the requested time is proportionnal to the number of GLL points, and to the number of source (receiver). Note : I erased the sqrt() in this part the source code (sqrt is positive growing function therefore it doesn't change the result), and I won 30% of calculation time (90s to 60s).

[specfem3d-zhang-ksu]

Thank you for your helpful comment, Etienne. I probably need seriously consider about how to speed this up because my receiver number is huge.

[EtienneBachmann]

I realized that my "trick" of suppressing sqrt() is in fact already used in locate_receivers routine (but not for sources). But I found even better for you : if you use the same mesh and the same receiver stations for various runs, you can turn on the flag "SU_FORMAT" in Par_file. You run one simulation, and at the end it will create you a file "SU_stations_info.bin". You don't suppress this file, and when you will run a new simulation, locate_receivers routine will read in this file all the required informations about the receivers, it should be really quicker.

[specfem3d-zhang-ksu]

This may be good for testing purpose. But my receiver configuration will change from run to run.

[komatits]

Thanks. I have now removed the sqrt() in the locate_source and locate_receivers routines in all three codes (2D, 3D and 3D_GLOBE).

Daniel has also improved other parts of these routines; they should now be significantly faster.

Dimitri.

On 03/19/2015 04:52 PM, EtienneBachmann wrote:

I totally agree with the fact that source and receiver location are the most time consuming stages before the timerun step. I try to turn on the flags that should accelerate this but I just won 1%. When looking at the source code, we can see the greedy time part (about 95%) : for each source (receiver), we loop on all the gll points of the MPI slice to check the distance to the source(receiver). There is probably an other way to do that, maybe with a Kd-Tree or something like this. I think this problem worths working on it, because the requested time is proportional to the number of GLL points, and to the number of source (receiver). Note : I erased the sqrt() in this part the source code (sqrt is positive growing function therefore it doesn't change the result), and I won 30% of calculation time (90s to 60s).

— Reply to this email directly or view it on GitHub https://github.com/geodynamics/specfem3d/issues/465#issuecomment-83641242.

Dimitri Komatitsch CNRS Research Director (DR CNRS), Laboratory of Mechanics and Acoustics, UPR 7051, Marseille, France http://komatitsch.free.fr

[komatits]

Thanks. Daniel and I have now improved that, and also removed all sqrt() statements in 2D, 3D and 3D_GLOBE.

Dimitri.

On 03/19/2015 09:22 PM, EtienneBachmann wrote:

I realized that my "trick" of suppressing sqrt() is in fact already used in locate_receivers routine (but not for sources). But I found even better for you : if you use the same mesh and the same receiver stations for various runs, you can turn on the flag "SU_FORMAT" in Par_file. You run one simulation, and at the end it will create you a file "SU_stations_info.bin". You don't suppress this file, and when you will run a new simulation, locate_receivers routine will read in this file all the required informations about the receivers, it should be really quicker.

— Reply to this email directly or view it on GitHub https://github.com/geodynamics/specfem3d/issues/465#issuecomment-83750671.

Dimitri Komatitsch CNRS Research Director (DR CNRS), Laboratory of Mechanics and Acoustics, UPR 7051, Marseille, France http://komatitsch.free.fr