Concurrency crashes in acoustic adapter

[jgroehl]

Encountered a crash running multiple Sims in parallel on Ubuntu, where a file in /tmp/xyz.mat was deleted by another process. Probably a shared file where it is highly unlikely but not impossible that the processes interact..

Will upload more details when I encounter this issue again.

[jgroehl]

2022-04-29 09:13:53,248 - INFO - Simulating the optical forward process...[Done]
2022-04-29 09:13:53,279 - DEBUG - Running <class 'simpa.core.simulation_modules.acoustic_forward_module.acoustic_forward_module_k_wave_adapter.KWaveAdapter'>
2022-04-29 09:13:53,279 - INFO - Simulating the acoustic forward process...
2022-04-29 09:13:53,279 - DEBUG - OPTICAL_PATH: /simulations/optical_forward_model_output/
2022-04-29 09:14:08,065 - DEBUG - field_of_view_extent: [-16  16   0   0 -16  16]
2022-04-29 09:14:08,072 - WARNING - The key for the Settings dictionary should be a tuple in the form of ('detector_element_width_mm', (data_type_1, data_type_2, ...)). The tuple of data types specifies all possible types, the value can have.
The key 'detector_element_width_mm' has been given the value 0.635
2022-04-29 09:14:08,072 - DEBUG - Added parameter ('voxel_spacing_mm', (<class 'int'>, <class 'numpy.integer'>, <class 'float'>)) to kWave settings via global_settings
2022-04-29 09:14:08,072 - WARNING - Did not find parameter ('model_sensor_frequency_response', <class 'bool'>) in any settings.
2022-04-29 09:14:08,072 - DEBUG - Added parameter ('medium_alpha_power', (<class 'int'>, <class 'numpy.integer'>, <class 'float'>)) to kWave settings via component_settings
2022-04-29 09:14:08,072 - DEBUG - Added parameter ('gpu', (<class 'bool'>, <class 'bool'>, <class 'numpy.bool_'>)) to kWave settings via global_settings
2022-04-29 09:14:08,072 - DEBUG - Added parameter ('pml_inside', <class 'bool'>) to kWave settings via component_settings
2022-04-29 09:14:08,072 - DEBUG - Added parameter ('pml_alpha', (<class 'int'>, <class 'numpy.integer'>, <class 'float'>)) to kWave settings via component_settings
2022-04-29 09:14:08,072 - DEBUG - Added parameter ('plot_pml', <class 'bool'>) to kWave settings via component_settings
2022-04-29 09:14:08,072 - DEBUG - Added parameter ('record_movie', (<class 'bool'>, <class 'bool'>, <class 'numpy.bool_'>)) to kWave settings via component_settings
2022-04-29 09:14:08,073 - DEBUG - Added parameter ('movie_name', <class 'str'>) to kWave settings via component_settings
2022-04-29 09:14:08,073 - DEBUG - Added parameter ('acoustic_log_scale', (<class 'bool'>, <class 'bool'>, <class 'numpy.bool_'>)) to kWave settings via component_settings
2022-04-29 09:14:08,073 - WARNING - Did not find parameter sensor_directivity_pattern in any settings.
2022-04-29 09:14:08,073 - WARNING - Did not find parameter ('initial_pressure_smoothing', <class 'bool'>) in any settings.
2022-04-29 09:14:08,147 - INFO - Simulating 2D....
2022-04-29 09:14:08,147 - INFO - ['/home/bohndieklab-cluster/matlab/bin/matlab', '-nodisplay', '-nosplash', '-automation', '-wait', '-r', "addpath('/home/bohndieklab-cluster/simpa/simpa/core/simulation_modules/acoustic_forward_module');simulate_2D('/home/bohndieklab-cluster/janek/simulations_3/Random_12000069.hdf5.mat');exit;"]

                            < M A T L A B (R) >
                  Copyright 1984-2021 The MathWorks, Inc.
             R2021b Update 3 (9.11.0.1873467) 64-bit (glnxa64)
                              February 3, 2022

Warning: Unrecognized command line option: automation. 
Warning: Unrecognized command line option: wait. 

To get started, type doc.
For product information, visit www.mathworks.com.

   847

   844

    0.3487

Setting custom time!
Running k-Wave simulation...
  start time: 29-Apr-2022 09:14:14
  reference sound speed: 1488m/s
Warning: Support for ver('distcomp') will be removed in a future release.  Use
ver('parallel') instead. 
> In ver>locGetSingleToolboxInfo (line 283)
In ver (line 56)
In verLessThan (line 39)
In kspaceFirstOrder_inputChecking (line 1306)
In kspaceFirstOrder2D (line 537)
In kspaceFirstOrder3DC (line 532)
In kspaceFirstOrder2DG (line 76)
In simulate_2D (line 172) 
  WARNING: visualisation plot scale may not be optimal for given source.
  dt: 25ns, t_end: 85.7us, time steps: 3429
  input grid size: 847 by 844 grid points (90.3467 by 90.0267mm)
  maximum supported frequency: 6.9668MHz by 6.975MHz
  smoothing p0 distribution...
  expanding computational grid...
  computational grid size: 875 by 864 grid points
  smoothing sound speed distribution...
  smoothing density distribution...
  precomputation completed in 0.96672s
  saving input files to disk...
Error using hdf5lib2
Unable to open '/tmp/kwave_input_data29-Apr-2022-09-14-14.h5'. File or folder
not found.

Error in H5F.open (line 130)
file_id = H5ML.hdf5lib2('H5Fopen', filename, flags, fapl, is_remote);

Error in h5writeatt (line 94)
fileId = H5F.open(Filename,'H5F_ACC_RDWR','H5P_DEFAULT');

Error in writeMatrix (line 208)
h5writeatt(filename, ['/' matrix_name], DATA_TYPE_ATT_NAME, data_type_c, 'TextEncoding', 'system');

Error in kspaceFirstOrder_saveToDisk (line 477)
        writeMatrix(flags.save_to_disk, eval(integer_variable_list{cast_index}), integer_variable_list{cast_index}, hdf_compression_level);

Error in kspaceFirstOrder2D (line 620)
    kspaceFirstOrder_saveToDisk;

Error in kspaceFirstOrder3DC (line 532)
eval(run_string);

Error in kspaceFirstOrder2DG (line 76)
sensor_data = kspaceFirstOrder3DC(varargin{:});

Error in simulate_2D (line 172)
    time_series_data = kspaceFirstOrder2DG(kgrid, medium, source, sensor, input_args{:});

Traceback (most recent call last):
  File "/home/bohndieklab-cluster/janek/palpaitine_simulation/simulate_random_phantoms_3.py", line 246, in <module>
    sp.simulate(simulation_pipeline=pipeline, digital_device_twin=device, settings=settings)
  File "/home/bohndieklab-cluster/simpa/simpa/core/simulation.py", line 78, in simulate
    pipeline_element.run(digital_device_twin)
  File "/home/bohndieklab-cluster/simpa/simpa/core/simulation_modules/acoustic_forward_module/__init__.py", line 67, in run
    time_series_data = self.forward_model(_device)
  File "/home/bohndieklab-cluster/simpa/simpa/core/simulation_modules/acoustic_forward_module/acoustic_forward_module_k_wave_adapter.py", line 118, in forward_model
    time_series_data, global_settings = self.k_wave_acoustic_forward_model(
  File "/home/bohndieklab-cluster/simpa/simpa/core/simulation_modules/acoustic_forward_module/acoustic_forward_module_k_wave_adapter.py", line 255, in k_wave_acoustic_forward_model
    raw_time_series_data = sio.loadmat(optical_path)[Tags.DATA_FIELD_TIME_SERIES_DATA]
KeyError: 'time_series_data'

[jgroehl]

Error using hdf5lib2 Unable to open '/tmp/kwave_input_data29-Apr-2022-09-14-14.h5'. File or folder not found.

Perhaps multiple files with exactly the same path name are being created sometimes?

[kdreher]

Does this also happen if you launch multiple simulations in quick succession from matlab? I think this is a k-wave problem because it seems to save temporary data in a file with a default name that uses a time stamp that is only accurate until a second. So I would guess if these tmp files are created in the same second, the same naming would apply for both files, no?

[jgroehl]

Super weird, haven't yet figured out why this is happening, but it keeps happening 🤦 Never had that before. Also just had a single simulation run that crashed the same way... Maybe it is an Ubuntu 21.04 issue?

[kdreher]

I never encountered this :sweat_smile: Maybe you could share a script and I can try to reproduce the error?