search

gadgetron / gadgetron

Gadgetron - Medical Image Reconstruction Framework
http://gadgetron.github.io/
Other
239 stars 164 forks source link

Storage Server fails to store the dependency #1224

Closed btasdelen closed 1 year ago

btasdelen commented 1 year ago

Describe the bug gadgetron_ismrmrd_client command fails at dependency storage with an error message on my system. If I use default_measurement_dependencies.xml instead of default_measurement_dependencies_storage.xml, the rest of the reconstruction succeeds, but of course dependencies are not stored and used.

I tried this with a variety of raw data, with the same result. Gadgetron installation on our remote server with Ubuntu installed works, so it feels like a configuration issue on my machine (Arch Linux) rather than Gadgetron issue. Still, any pointers would be appreciated, as I would like to work on my local machine.

To Reproduce Steps to reproduce the behavior, e.g.:

  1. Start with clean gadgetron conda environment.
  2. Start the gadgetron server.
  3. Run gadgetron_ismrmrd_client --verbose -f noise_meas.h5 -c default_measurement_dependencies_ismrmrd_storage.xml
  4. See error: Client side: 
    Gadgetron ISMRMRD client
-- host            :      localhost
-- port            :      9021
-- hdf5 file  in   :      /mnt/LIN_DATA/MRI_DATA/vol0160/raw/noise/noise_meas_MID00049_FID20554_t1_vibe_dixon_963_bh_3_echos_arms_up.h5
-- hdf5 group in   :      /dataset
-- conf            :      default_measurement_dependencies_ismrmrd_storage.xml
-- loop            :      1
-- hdf5 file out   :      out.h5
-- hdf5 group out  :      2023-06-18 17:37:01
Find 1842 ismrmrd acquisitions
Find 0 ismrmrd waveforms
terminate called after throwing an instance of 'boost::wrapexcept<boost::system::system_error>'
what():  read: End of file [asio.misc:2]
Error caught: send_ismrmrd_acq failed ... [1]    24569 IOT instruction (core dumped)  gadgetron_ismrmrd_client --verbose -p 9021 -f  -c

Server side:

06-18 17:36:58.194 DEBUG [gadgetron_paths.cpp:117] Executable path: "/mnt/LIN_DATA/anaconda3/envs/gadgetron2/bin/gadgetron"
06-18 17:36:58.194 DEBUG [gadgetron_paths.cpp:123] Default Gadgetron home: "/mnt/LIN_DATA/anaconda3/envs/gadgetron2"
06-18 17:36:58.194 WARNING [initialization.cpp:38] Environment variable 'OMP_WAIT_POLICY' not set to 'PASSIVE'.
06-18 17:36:58.194 WARNING [initialization.cpp:39] Gadgetron may experience serious performance issues under heavy load (multiple simultaneous reconstructions, etc.)
06-18 17:36:58.195 INFO [main.cpp:90] Gadgetron 4.4.7 [a0318d1349dce1b9799cb9f6b4486c6cfc5e0c7f]
06-18 17:36:58.195 DEBUG [storage.cpp:68] Found storage server: /mnt/LIN_DATA/anaconda3/envs/gadgetron2/bin/mrd-storage-server
06-18 17:36:58.195 INFO [storage.cpp:69] Starting storage server on port 9131
06-18 17:36:58.196 INFO [storage.cpp:28] Verifying connectivity to storage server...
{"level":"info","time":"2023-06-18T17:36:58.201-07:00","message":"Listening on port 9131"}
{"level":"info","requestId":"bb622816-f864-45a1-be9a-9ef2411f5b08","status":200,"method":"GET","path":"/healthcheck","query":"","latencyMs":0.799,"time":"2023-06-18T17:36:58.398-07:00","message":"request completed"}
06-18 17:36:58.399 INFO [storage.cpp:33] Received successful response from storage server.
06-18 17:36:58.399 INFO [main.cpp:99] Running on port 9021
06-18 17:36:58.399 INFO [Server.cpp:25] Gadgetron home directory: "/mnt/LIN_DATA/anaconda3/envs/gadgetron2"
06-18 17:36:58.399 INFO [Server.cpp:26] Gadgetron working directory: "/tmp/gadgetron/"
06-18 17:37:01.229 INFO [Server.cpp:38] Accepted connection from: ::ffff:127.0.0.1
06-18 17:37:01.230 INFO [ConfigConnection.cpp:131] Connection state: [CONFIG]
06-18 17:37:01.230 DEBUG [ConfigConnection.cpp:73] Reading config file: "/mnt/LIN_DATA/anaconda3/envs/gadgetron2/share/gadgetron/config/default_measurement_dependencies_ismrmrd_storage.xml"
06-18 17:37:01.231 INFO [HeaderConnection.cpp:84] Connection state: [HEADER]
06-18 17:37:01.232 INFO [StreamConnection.cpp:75] Connection state: [STREAM]
06-18 17:37:01.242 DEBUG [Stream.cpp:58] Loading Gadget IsmrmrdDump of class IsmrmrdDumpGadget from gadgetron_mricore
06-18 17:37:01.242 DEBUG [Stream.cpp:58] Loading Gadget NoiseAdjust of class NoiseAdjustGadget from gadgetron_mricore
06-18 17:37:01.242 DEBUG [IsmrmrdDumpGadget.cpp:36] IsmrmrdDumpGadget, find pre-set ip for no-data-saving : 2 [ 
06-18 17:37:01.242 DEBUG [IsmrmrdDumpGadget.cpp:38] 192.168.2.2
06-18 17:37:01.242 DEBUG [IsmrmrdDumpGadget.cpp:38] 192.168.56.2
06-18 17:37:01.242 DEBUG [IsmrmrdDumpGadget.cpp:39]  ] 
06-18 17:37:01.242 DEBUG [NoiseAdjustGadget.cpp:195] Folder to store noise dependencies is /tmp/gadgetron
06-18 17:37:01.242 DEBUG [NoiseAdjustGadget.cpp:196] NoiseAdjustGadget::perform_noise_adjust_ is 1
06-18 17:37:01.242 DEBUG [NoiseAdjustGadget.cpp:197] NoiseAdjustGadget::pass_nonconformant_data_ is 1
06-18 17:37:01.242 DEBUG [NoiseAdjustGadget.cpp:198] receiver_noise_bandwidth_ is 0.793000
  1. Any subsequent call of the gadgetron_ismrmrd_client yields the following on the client side:
Gadgetron ISMRMRD client
  -- host            :      localhost
  -- port            :      9021
  -- hdf5 file  in   :      /mnt/LIN_DATA/MRI_DATA/vol0160/raw/noise/noise_meas_MID00049_FID20554_t1_vibe_dixon_963_bh_3_echos_arms_up.h5
  -- hdf5 group in   :      /dataset
  -- conf            :      default_measurement_dependencies_ismrmrd_storage.xml
  -- loop            :      1
  -- hdf5 file out   :      out.h5
  -- hdf5 group out  :      2023-06-18 17:50:28
Find 1842 ismrmrd acquisitions
Find 0 ismrmrd waveforms
terminate called after throwing an instance of 'boost::wrapexcept<boost::system::system_error>'
  what():  read: Connection reset by peer [system:104]
Error caught: [1]    25155 IOT instruction (core dumped)  gadgetron_ismrmrd_client --verbose -p 9021 -f  -c

Expected behavior Measurement dependency to be stored.

System Information:

hansenms commented 1 year ago

I think this is probably a bit of a misunderstanding based on the poor naming of the those configurations. Both of them store the noise covariance in the storage server, but the default_measurement_dependencies_ismrmrd_storage.xml also stores the raw data on disk. It has the IsmrmrdDump gadget in the pipeline as well. The reason why that one is failing might be because it can't write to the disk location where it is trying to write, but it could be some other issue.