Have some problems when trying to create an Example for Seisflows based on Marmousi data.

[zhangxiaoshuotttt]

Hi @bch0w , I am trying to write some examples for Seisflows to help beginners like me learn more about Seisflows. Now, I'm trying to use this marmousi example. And I tried to I tried to draw on this document. Only now have I encountered some problems.

Now, I hope to list these problems that may help us make Seisflows better. (I do love Seisflows, it's convenient and powerful :）

1.

Here is the problem when I tried to perform a serial inversion: If we want to follow this tutorial. We need to set generate_data true; otherwise, we don't have the data in the folder data to perform the inversion.

If we don't set this ! seisflows par generate_data true Then

////////////////////////////////////////////////////////////////////////////////
                              RUNNING ITERATION 01                              
////////////////////////////////////////////////////////////////////////////////
2024-04-01 21:50:32 [INFO] | 
================================================================================
                           RUNNING INVERSION WORKFLOW                           
================================================================================
2024-04-01 21:50:32 [INFO] | 
////////////////////////////////////////////////////////////////////////////////
                      EVALUATING MISFIT FOR INITIAL MODEL                       
////////////////////////////////////////////////////////////////////////////////
2024-04-01 21:50:32 [INFO] | evaluating misfit for model in `path_model_init`
2024-04-01 21:50:32 [INFO] | checking model parameters:
2024-04-01 21:50:32 [INFO] | vp: min=1500.000; mean=2803.176; max=3955.629
2024-04-01 21:50:32 [INFO] | vs: min=0.000E+00; mean=1.293E+03; max=2.186E+03
2024-04-01 21:50:32 [INFO] | preparing observation data for source 001
2024-04-01 21:50:32 [DEBU] | looking for data in: '/home/master/seisflows_workdir/Marmousi/data/001/*'
2024-04-01 21:50:32 [CRIT] | 
================================================================================
                               DATA IMPORT ERROR                                
                               /////////////////                                
001 found no `obs` data with wildcard:
'/home/master/seisflows_workdir/Marmousi/data/001/*'. Please check `path_data`
or manually import data and re-submit
================================================================================

2.

Here is the problem when I tried to perform a parallel version. This is a question that I'm not sure. When I use mpirun to generate OUTPUT_FILES_TRUE and OUTPUT_FILES_INIT, you know We will get a lot of files, for example, the velocity file. We only get one file when performing a serial version, like proc000000_vp.bin, but if we perform a parallel version, we get proc000000_vp.bin,proc000001_vp.bin,...Then, if we continue, the Seisflows will tell us that we have too many model files(This is not exactly, but you can have a test). Until now, I'm trying to figure out how to perform a parallel inversion. Can you give me some advice? I also want to ask, in Seisflows, can we only read one model file(like proc000000_vp.bin)? Here are details.

# GENERATE MODEL_INIT
**##########Setting when preparing initial(true) files################**
os.chdir(SPECFEM2D_WORKDIR)

# Run the mesher and solver to generate our initial model
! mpirun -np 12 ./bin/xmeshfem2D > OUTPUT_FILES/mesher_log.txt
! mpirun -np 12 ./bin/xspecfem2D > OUTPUT_FILES/solver_log.txt

#! ./bin/xmeshfem2D > OUTPUT_FILES/mesher_log.txt
#! ./bin/xspecfem2D > OUTPUT_FILES/solver_log.txt

# Move the model files (*.bin) into the OUTPUT_FILES directory, where SeisFlows3 expects them
! mv DATA/*bin OUTPUT_FILES

# Make sure we don't overwrite this initial model when creating our target model in the next step
! mv OUTPUT_FILES OUTPUT_FILES_TRUE

! head OUTPUT_FILES_TRUE/solver_log.txt
! tail OUTPUT_FILES_TRUE/solver_log.txt
**##########Setting when preparing initial(true) files################**

////////////////////////////////////////////////////////////////////////////////
                              RUNNING ITERATION 01                              
////////////////////////////////////////////////////////////////////////////////
2024-04-01 21:44:54 [INFO] | 
================================================================================
                           RUNNING INVERSION WORKFLOW                           
================================================================================
2024-04-01 21:44:54 [INFO] | 
////////////////////////////////////////////////////////////////////////////////
                   GENERATING SYNTHETIC DATA W/ TARGET MODEL                    
////////////////////////////////////////////////////////////////////////////////
2024-04-01 21:44:54 [INFO] | checking true/target model parameters:
Traceback (most recent call last):
  File "/home/master/anaconda3/envs/seisflows/bin/seisflows", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/master/seisflows/seisflows/seisflows.py", line 1428, in main
    sf()
  File "/home/master/seisflows/seisflows/seisflows.py", line 450, in __call__
    getattr(self, self._args.command)(**vars(self._args))
  File "/home/master/seisflows/seisflows/seisflows.py", line 802, in submit
    system.submit(workdir=self._args.workdir,
  File "/home/master/seisflows/seisflows/system/workstation.py", line 211, in submit
    workflow.run()
  File "/home/master/seisflows/seisflows/workflow/inversion.py", line 198, in run
    super().run()  # Runs task list
    ^^^^^^^^^^^^^
  File "/home/master/seisflows/seisflows/workflow/forward.py", line 280, in run
    func()
  File "/home/master/seisflows/seisflows/workflow/inversion.py", line 237, in generate_synthetic_data
    super().generate_synthetic_data(**kwargs)
  File "/home/master/seisflows/seisflows/workflow/forward.py", line 310, in generate_synthetic_data
    self.solver.check_model_values(path=self.path.model_true)
  File "/home/master/seisflows/seisflows/solver/specfem.py", line 306, in check_model_values
    _model = Model(path=path, parameters=self._parameters,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/seisflows/seisflows/tools/model.py", line 113, in __init__
    self.model = self.read(parameters=parameters)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/seisflows/seisflows/tools/model.py", line 249, in read
    parameter_dict[parameter] = load_fx(parameter=parameter)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/seisflows/seisflows/tools/model.py", line 739, in _read_model_fortran_binary
    array = np.array(array)
            ^^^^^^^^^^^^^^^
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (12,) + inhomogeneous part.

3.

Seisflows reported that the 'su' format is not implemented for inversion.

4.

By the way, I have another question; I've seen an excellent example for modifying model. It was made by Gamsh. So, can we use this kind of SPECFEM example to perform inversion? I mean, this kind of example uses external meshes. So, can we use them to perform inversion? Do we need other settings?(Hi @mnagaso)

Forgive that this description may sound confusing. I will give my notebook as soon as possible after I finish my graduate thesis.

I hope this can help us make Seisflows better.

Warm regards,

Thanks again.

[zhangxiaoshuotttt]

Here, I found another problem. When I change the setting to ! seisflows par stop_after finalize_iteration (line 53 in tutorial notebook) I encounter this problem. This is because the initial model is not very different from the real model, right?

2024-04-04 00:37:50 [INFO] | `workflow.iteration` == 1
2024-04-04 00:37:50 [INFO] | `workflow.stop_after` == finalize_iteration
2024-04-04 00:37:50 [INFO] | setup Inversion workflow
2024-04-04 00:37:59 [DEBU] | running setup for module 'system.Workstation'
2024-04-04 00:38:00 [DEBU] | running setup for module 'solver.Specfem2D'
2024-04-04 00:38:00 [DEBU] | running setup for module 'preprocess.Default'
2024-04-04 00:38:01 [DEBU] | running setup for module 'optimize.Gradient'
2024-04-04 00:38:01 [INFO] | re-loading optimization module from checkpoint
2024-04-04 00:38:04 [INFO] | 
////////////////////////////////////////////////////////////////////////////////
                              RUNNING ITERATION 01                              
////////////////////////////////////////////////////////////////////////////////
2024-04-04 00:38:04 [INFO] | 
================================================================================
                           RUNNING INVERSION WORKFLOW                           
================================================================================
2024-04-04 00:38:04 [INFO] | 'generate_synthetic_data' has already been run, skipping
2024-04-04 00:38:04 [INFO] | 'evaluate_initial_misfit' has already been run, skipping
2024-04-04 00:38:04 [INFO] | 'run_adjoint_simulations' has already been run, skipping
2024-04-04 00:38:04 [INFO] | 'postprocess_event_kernels' has already been run, skipping
2024-04-04 00:38:04 [INFO] | scaling gradient to absolute model perturbations
2024-04-04 00:38:04 [WARN] | no coordinates found for assumed SPECFEM2D model, will not be able to plot figures
2024-04-04 00:38:06 [WARN] | no coordinates found for assumed SPECFEM2D model, will not be able to plot figures
2024-04-04 00:38:07 [INFO] | exporting gradient to disk
2024-04-04 00:38:07 [CRIT] | 
================================================================================
                          OPTIMIZATION GRADIENT ERROR                           
                          ///////////////////////////                           
Gradient vector 'g' is 0, meaning none of the kernels from the adjoint
simulations returned. Please check your gradient and waveform misfits.

[zhangxiaoshuotttt]

When I tried to use the Marmousi model to create an example for Seisflows. I encountered the same problem at this step. Here are all my settings. Regarding the Marmousi model, we know there are differences between the initial and actual waveforms. Please excuse the disorganized state of my notebook; I replaced the model of the example with Marmousi.

[zhangxiaoshuotttt]

I mistakenly used the development version instead of the stable Master version. Switching to the master version solved all errors. I hope these lists that could help with debugging.

[zhangxiaoshuotttt]

@bch0w Hi, and sorry to bother you again, I encounter a new problem with the example. Now I'd like to set "This." Then, I tried to perform the following: I found the g is zero. What may cause this? By the way, it always tells me it couldn't find the obs or syn data. Here are all my settings.

# ==============================================================================
workflow: inversion
system: workstation
solver: specfem2d
preprocess: default
optimize: LBFGS

data_case: data
stop_after: evaluate_initial_misfit
export_traces: True
export_residuals: False
export_gradient: True
export_kernels: True
start: 1
end: 3
export_model: True
thrifty: False
iteration: 1

ntask: 3
nproc: 1
mpiexec: null
log_level: DEBUG
verbose: False

syn_data_format: ascii
materials: elastic
density: False
attenuation: False
smooth_h: 0.0
smooth_v: 0.0
components: ZX
source_prefix: SOURCE
multiples: True

obs_data_format: ascii
unit_output: VEL
misfit: waveform
adjoint: waveform
normalize: []
filter: null
min_period: null
max_period: null
min_freq: null
max_freq: null
mute: []
early_slope: null
early_const: null
late_slope: null
late_const: null
short_dist: null
long_dist: null

preconditioner: null
step_count_max: 10
step_len_init: 0.05
step_len_max: 0.5
line_search_method: Backtrack
LBFGS_mem: 3
LBFGS_max: inf
LBFGS_thresh: 0.0

path_workdir: .
path_scratch: scratch
path_eval_grad: scratch/eval_grad
path_output: output
path_model_init: /home/master/seisflows_workdir/MarmousiField/specfem2d_workdir/OUTPUT_FILES_INIT
path_model_true: null
path_state_file: sfstate.txt
path_data: /home/master/seisflows_workdir/MarmousiField/specfem2d_workdir/FieldData
path_mask: null
path_eval_func: scratch/eval_func
path_par_file: parameters.yaml
path_log_files: logs
path_output_log: sflog.txt
path_specfem_bin: /home/master/seisflows_workdir/MarmousiField/specfem2d_workdir/bin
path_specfem_data: /home/master/seisflows_workdir/MarmousiField/specfem2d_workdir/DATA
path_solver: scratch/solver
path_preconditioner: null

And here are the names of my obs data(generated by Specfem2d) Here are errors:

////////////////////////////////////////////////////////////////////////////////
                              RUNNING ITERATION 01                              
////////////////////////////////////////////////////////////////////////////////
2024-04-04 22:58:11 (I) | 
================================================================================
                           RUNNING INVERSION WORKFLOW                           
================================================================================
2024-04-04 22:58:11 (I) | 
////////////////////////////////////////////////////////////////////////////////
                      EVALUATING MISFIT FOR INITIAL MODEL                       
////////////////////////////////////////////////////////////////////////////////
2024-04-04 22:58:11 (I) | checking initial model parameters
2024-04-04 22:58:11 (I) | 1500.00 <= vp <= 3919.63
2024-04-04 22:58:11 (I) | 0.00E+00 <= vs <= 2.18E+03
2024-04-04 22:58:11 (I) | preparing observation data for source 001
2024-04-04 22:58:11 (I) | copying data from `path_data`
2024-04-04 22:58:11 (I) | evaluating objective function for source 001
2024-04-04 22:58:11 (D) | running forward simulation with 'Specfem2D'
2024-04-04 22:58:11 (D) | running executable with cmd: 'bin/xmeshfem2D'
2024-04-04 22:58:13 (D) | running executable with cmd: 'bin/xspecfem2D'
2024-04-04 22:58:43 (D) | quantifying misfit with 'Default'
Traceback (most recent call last):
  File "/home/master/anaconda3/envs/seisflows/bin/seisflows", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/master/seisflows/seisflows/seisflows.py", line 1395, in main
    sf()
  File "/home/master/seisflows/seisflows/seisflows.py", line 447, in __call__
    getattr(self, self._args.command)(**vars(self._args))
  File "/home/master/seisflows/seisflows/seisflows.py", line 704, in submit
    system.submit(workdir=self._args.workdir,
  File "/home/master/seisflows/seisflows/system/workstation.py", line 172, in submit
    workflow.run()
  File "/home/master/seisflows/seisflows/workflow/inversion.py", line 184, in run
    super().run()  # Runs task list
    ^^^^^^^^^^^^^
  File "/home/master/seisflows/seisflows/workflow/forward.py", line 287, in run
    func()
  File "/home/master/seisflows/seisflows/workflow/inversion.py", line 272, in evaluate_initial_misfit
    super().evaluate_initial_misfit()
  File "/home/master/seisflows/seisflows/workflow/forward.py", line 343, in evaluate_initial_misfit
    self.system.run(run_list, path_model=self.path.model_init,
  File "/home/master/seisflows/seisflows/system/workstation.py", line 206, in run
    func(**kwargs)
  File "/home/master/seisflows/seisflows/workflow/inversion.py", line 237, in evaluate_objective_function
    self.preprocess.quantify_misfit(
  File "/home/master/seisflows/seisflows/preprocess/default.py", line 510, in quantify_misfit
    observed, synthetic = self._setup_quantify_misfit(source_name)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/seisflows/seisflows/preprocess/default.py", line 422, in _setup_quantify_misfit
    assert(len(observed) != 0 and len(synthetic) != 0), \
                                  ^^^^^^^^^^^^^^^^^^^
AssertionError: cannot quantify misfit, missing observed or synthetic traces

[zhangxiaoshuotttt]

I'm not sure if this code can copy the field data into the obs folder.

[zhangxiaoshuotttt]

1. The waveform of syn

I then manually moved the field data into the obs folder. But after performing seisflows submit, the obs became the same as the syn. It's wired. I've struggled with this problem for four days.

2. The original field data

3. After submit, the waveform inside the obs become as follows

[bch0w]

Hi @zhangxiaoshuotttt, sorry for the late reply here, and sorry to hear you're having issues with this.

From what I understand your current problem is that you are not able to get your obs data read in for the inversion? One note is that your obs data should be discoverable in the following hard-coded directory structure:

path_data/<SOURCE_NAME>/*

Where path_data is defined in your parameters.yaml file, and should match the suffixes for your sources. I think in your case it would be 001, 002, 003. As a double check they should also match the directory names in scratch/solver.

Also you should be aware of the parameter obs_data_format which defines the expected format of your observed data. Since you have it listed as ascii, SeisFlows will be looking for SPECFEM generated waveform files.

Hopefully one of those helps your problem, let me know if you're still having issues.

[bch0w]

Closing for inactivity, please open a new issue if you are still facing trouble!