adjtomo / seisflows

An automated workflow tool for full waveform inversion and adjoint tomography
http://seisflows.readthedocs.org
BSD 2-Clause "Simplified" License
178 stars 122 forks source link

IndexError while running an acoustic example #182

Closed raulleoncz closed 6 months ago

raulleoncz commented 8 months ago

Hello everyone. I've been trying to run an acoustic inversion problem using specfem2d as my solver and 'su' as the data format but I'm encountering the same error. Below there's a ss of the error. Screenshot from 2024-01-18 12-57-03

I've trying with different parameters thinking that maybe I was forgetting something but I got the same error. Here are some of the parameters that I'm using:

workflow: inversion data_case: synthetic stop_after: evaluate_initial_misfit syn_data_format: su materials: acoustic obs_data_format: su stop misfit: traveltime

I hope you can help me to understand what I'm doing wrong. Thank you.

bch0w commented 8 months ago

Hi @raulleoncz, thanks for raising this issue. At the moment the SU data format is underdeveloped with respect to the ASCII format option, and I suspect that may the source of your issue (see related issue: https://github.com/adjtomo/seisflows/issues/174). I have had it on my to do list to bugfix the SU capabilities but have not had the time.

Can you try setting the data format to ASCII and seeing if you run into issues? Also if you could provide your parameter.yaml file, that might help me recreate the issue and see if I can put out a fix to get you past this error.

raulleoncz commented 8 months ago

Thank you so much for the quick answer! Ok. I ran the same simulation following your advice and it worked. It appears that the problem could be related to what you already said is a bug or maybe related to the components I specified. The parameters that I used for the ascii format are these:

Modules: workflow: inversion system: workstation solver: specfem2d preprocess: default optimize: gradient

Forward and Inversion: data_case: synthetic stop_after: evaluate_initial_misfit export_traces: False export_residuals: False export_gradient: True export_kernels: False start: 1 end: 2 export_model: True thrifty: False iteration: 1

Workstation: ntask: 1 nproc: 1 mpiexec: null log_level: DEBUG verbose: False

Solver: syn_data_format: ascii materials: acoustic density: False attenuation: False smooth_h: 0.0 smooth_v: 0.0 components: Z source_prefix: SOURCE multiples: False

Preprocess: obs_data_format: ascii unit_output: disp misfit: traveltime adjoint: traveltime normalize: [] filter: null min_period: null max_period: null min_freq: null max_freq: null mute: [] early_slope: null early_const: null late_slope: null late_const: null short_dist: null long_dist: null

Optimization: preconditioner: null step_count_max: 5 step_len_init: 0.05 step_len_max: 0.5 line_search_method: bracket

Paths: path_model_init: /home/rleon/Example/specfem2d_workdir/OUTPUT_FILES_INIT path_model_true: /home/rleon/Example/specfem2d_workdir/OUTPUT_FILES_TRUE path_specfem_bin: /home/rleon/Example/specfem2d_workdir/bin path_specfem_data: /home/rleon/Example/specfem2d_workdir/DATA

For the simulation using SU, I just changed the syn and obs format to 'su' and components to 'd', this because I was following the structure of the output files 'Ux_file_single_d' (maybe this could be my error). Thank you again for your help.

bch0w commented 8 months ago

Thanks for the update @raulleoncz, glad that ASCII format works for you. Yes this is likely coming from the underdeveloped SU format option, and I think in your specific case you are correct that the filenames incorrectly searched for by SeisFlows. I hope I can carve out some time soon to work through this as it looks to be a useful feature for many.

raulleoncz commented 8 months ago

Update. Hello @bch0w, sorry for the inconvenience. I've been trying to run the next step of the workflow which should stop after evaluating the gradient from kernels but the process had been failed at the last part "Generating/Processing Misfit Kernel". At the beginning I though it was something related to my specfem version which seems to have problems reading tomo files but I ran the same simulation on a cluster with a version that works well I still got the same error. I show you the error in the following images: Screenshot from 2024-01-23 11-47-31 Screenshot from 2024-01-23 11-36-57

I checked the misfit folder and it is indeed empty. The parameters that I used are practically the same, obviously I change the system to cluster and I specified mpiexec as mpirun. I hope you can help me. Thank you.

bch0w commented 8 months ago

Hi @raulleoncz, missing kernels might be explained by a few different things so we might have to poke around to find the issue. I would check that your adjoint simulations are running successfully, and if they are, check that your adjoint sources are present.

To look at the adjoint simulations you will have some log files named something like: scratch/solver/mainsolver/adj_solver.log scratch/solver/mainsolver/OUTPUT_FILES/output_solver.txt The first should be empty, and the second should indicate that the adjoint simulation ran successfully

If those do not show any issue, then perhaps the adjoint sources in: scratch/solver/mainsolver/traces/adj There should be a number of traces in there, equal to the amount of synthetic waveforms you have in syn.

If that is also alright, then let me know and I can suggest some other places to look!

P.S. if we need to troubleshoot some more I may ask you to start a new issue just to keep a separate record

bch0w commented 6 months ago

Hi @raulleoncz, I am closing this issue as I think we migrated the discussion to #185 and now to #202. Please re-open if I am mistaken, thanks