search

SPECFEM / specfem3d

SPECFEM3D_Cartesian simulates acoustic (fluid), elastic (solid), coupled acoustic/elastic, poroelastic or seismic wave propagation in any type of conforming mesh of hexahedra (structured or not).
GNU General Public License v3.0
389 stars 223 forks source link

Error running adjoint simulations #1680

Closed aakash10gupta closed 3 months ago

aakash10gupta commented 4 months ago

Description

Error invalid irec for local adjoint source

I am receiving the above error while trying to run adjoint simulations using the latest master branch. The forward simulation runs fine but the adjoint simulation does not. The output_solver.txt file has no error messages but stops being written out midway. The cluster output log displays the above error message.

After some debugging it looks like the error is being caused by changes to the file setup_sources_receivers.f90 in the commit 5530249. I was able to run adjoint simulations with a commit just prior to this suspected one.

bf45798 - latest master (FAILS) 5530249 - suspected commit (NOT TESTED) 172f4ec - commit prior to the suspected one (SUCCEEDS)

Files to reproduce the error using the steps mentioned below - kernels.zip

Affected SPECFEM3D version

bf45798f3af9d792326a829de920fd944cf7c7dd (and probably some prior commits as well)

Your software and hardware environment

HPC cluster - chinook04 (UAF) icc (ICC) 2021.6.0 20220226 ifort (IFORT) 2021.6.0 20220226

Reproduction steps

  1. Clone latest specfem master branch bf45798 and compile using make all.
  2. Also compile create_adjsrc_traveltime.f90 by running make in its respective folder, and move the thus created xcreate_adjsrc_traveltime to the bin folder.
  3. Unzip the above attached zipped folder.
  4. Copy compiled specfem bin folder into the unzipped folder.
  5. Run the provided kernels.bash script on a slurm managed cluster or otherwise.
  6. Wait for the job to fail and see error.

Logs

Error invalid irec for local adjoint source

OS

Rocky Linux 8.7

danielpeter commented 3 months ago

good catch, thanks Aakash! should be addressed by PR #1686 - please doublecheck with your example again.

aakash10gupta commented 3 months ago

Thanks Daniel! I doublechecked and it's working fine now.