search

nipy / heudiconv

Flexible DICOM conversion into structured directory layouts
https://heudiconv.readthedocs.io
Other
238 stars 126 forks source link

pydicom warnings encountered during `test_b0dwi_for_fmap` #667

Open jwodder opened 1 year ago

jwodder commented 1 year ago

When the test_b0dwi_for_fmap test in heudiconv/tests/test_convert.py is run, pydicom emits two UserWarnings with the message "The value length (22) exceeds the maximum length of 16 allowed for VR SH." (presumably because one or more files in heudiconv/tests/data/b0dwiForFmap/ are ill-formed), and then pytest's filterwarnings setting causes these warnings to be converted into errors that are caught & logged here — and yet, the test does not fail.

For the record, the output from running just this test as of commit 2c6d228 is:

``` ============================= test session starts ============================== platform darwin -- Python 3.11.3, pytest-7.3.0, pluggy-1.0.0 -- /Users/jwodder/dartmouth/heudiconv/.tox/py3/bin/python cachedir: .tox/py3/.pytest_cache rootdir: /Users/jwodder/dartmouth/heudiconv configfile: tox.ini collecting ... collected 120 items / 119 deselected / 1 selected heudiconv/tests/test_convert.py::test_b0dwi_for_fmap WARNING: Heuristic is missing an `infotoids` method, assigning empty method and using provided subject id b0dwiForFmap. Provide `session` and `locator` fields for best results. 230412-10:38:52,754 nipype.workflow INFO: [Node] Setting-up "convert" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/dcm2niix25vaxluk/convert". INFO: [Node] Setting-up "convert" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/dcm2niix25vaxluk/convert". 230412-10:38:52,762 nipype.workflow INFO: [Node] Executing "convert" INFO: [Node] Executing "convert" 230412-10:38:52,949 nipype.interface INFO: stdout 2023-04-12T10:38:52.949521:Compression will be faster with 'pigz' installed http://macappstore.org/pigz/ INFO: stdout 2023-04-12T10:38:52.949521:Compression will be faster with 'pigz' installed http://macappstore.org/pigz/ 230412-10:38:52,950 nipype.interface INFO: stdout 2023-04-12T10:38:52.949521:Chris Rorden's dcm2niiX version v1.0.20220720 Clang13.0.0 x86-64 (64-bit MacOS) 230412-10:38:52,950 nipype.interface INFO: stdout 2023-04-12T10:38:52.949521:Found 3 DICOM file(s) 230412-10:38:52,950 nipype.interface INFO: stdout 2023-04-12T10:38:52.949521:Convert 3 DICOM as /private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/pytest-of-jwodder/pytest-256/test_b0dwi_for_fmap0/sub-b0dwiForFmap/fmap/sub-b0dwiForFmap_acq-b0dwi_epi_heudiconv021 (150x152x81x3) INFO: stdout 2023-04-12T10:38:52.949521:Chris Rorden's dcm2niiX version v1.0.20220720 Clang13.0.0 x86-64 (64-bit MacOS) INFO: stdout 2023-04-12T10:38:52.949521:Found 3 DICOM file(s) INFO: stdout 2023-04-12T10:38:52.949521:Convert 3 DICOM as /private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/pytest-of-jwodder/pytest-256/test_b0dwi_for_fmap0/sub-b0dwiForFmap/fmap/sub-b0dwiForFmap_acq-b0dwi_epi_heudiconv021 (150x152x81x3) 230412-10:38:53,606 nipype.interface INFO: stdout 2023-04-12T10:38:53.606512:Conversion required 0.716647 seconds (0.715186 for core code). INFO: stdout 2023-04-12T10:38:53.606512:Conversion required 0.716647 seconds (0.715186 for core code). 230412-10:38:53,642 nipype.workflow INFO: [Node] Finished "convert", elapsed time 0.78719s. INFO: [Node] Finished "convert", elapsed time 0.78719s. WARNING: Diffusion-weighted image saved in non dwi folder (/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/pytest-of-jwodder/pytest-256/test_b0dwi_for_fmap0/sub-b0dwiForFmap/fmap) WARNING: .bvec and .bval files will be generated. This is NOT BIDS compliant 230412-10:38:53,660 nipype.workflow INFO: [Node] Setting-up "embedder" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmeta3r_3optu/embedder". INFO: [Node] Setting-up "embedder" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmeta3r_3optu/embedder". 230412-10:38:53,665 nipype.workflow INFO: [Node] Executing "embedder" INFO: [Node] Executing "embedder" 230412-10:38:53,689 nipype.workflow INFO: [Node] Finished "embedder", elapsed time 0.021059s. 230412-10:38:53,689 nipype.workflow WARNING: Storing result file without outputs INFO: [Node] Finished "embedder", elapsed time 0.021059s. WARNING: Storing result file without outputs 230412-10:38:53,690 nipype.workflow WARNING: [Node] Error on "embedder" (/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmeta3r_3optu/embedder) WARNING: [Node] Error on "embedder" (/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmeta3r_3optu/embedder) ERROR: Embedding failed: Exception raised while executing Node embedder. Traceback: Traceback (most recent call last): File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/tag.py", line 28, in tag_in_exception yield File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 2389, in walk data_element = self[tag] ~~~~^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 939, in __getitem__ self[tag] = DataElement_from_raw(elem, character_set, self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataelem.py", line 859, in DataElement_from_raw value = convert_value(vr, raw, encoding) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 768, in convert_value return converter(byte_string, encodings, VR) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in convert_text as_strings = [convert_single_string(value, encodings, vr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in as_strings = [convert_single_string(value, encodings, vr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 557, in convert_single_string validate_value( File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/valuerep.py", line 290, in validate_value warnings.warn(msg) UserWarning: The value length (22) exceeds the maximum length of 16 allowed for VR SH. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run runtime = self._run_interface(runtime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface out = function_handle(**args) ^^^^^^^^^^^^^^^^^^^^^^^ File "", line 30, in embed_dicom_and_nifti_metadata File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/dcmstack/dcmstack.py", line 1156, in parse_and_stack results = parse_and_group(src_paths, ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/dcmstack/dcmstack.py", line 1060, in parse_and_group meta = extractor(dcm) ^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/dcmstack/extract.py", line 455, in __call__ dcm.decode() File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 551, in decode self.walk(decode_callback, recursive=False) File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 2388, in walk with tag_in_exception(tag): File "/usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 155, in __exit__ self.gen.throw(typ, value, traceback) File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/tag.py", line 32, in tag_in_exception raise type(exc)(msg) from exc UserWarning: With tag (0051, 100e) got exception: The value length (22) exceeds the maximum length of 16 allowed for VR SH. Traceback (most recent call last): File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/tag.py", line 28, in tag_in_exception yield File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 2389, in walk data_element = self[tag] ~~~~^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 939, in __getitem__ self[tag] = DataElement_from_raw(elem, character_set, self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataelem.py", line 859, in DataElement_from_raw value = convert_value(vr, raw, encoding) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 768, in convert_value return converter(byte_string, encodings, VR) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in convert_text as_strings = [convert_single_string(value, encodings, vr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in as_strings = [convert_single_string(value, encodings, vr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 557, in convert_single_string validate_value( File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/valuerep.py", line 290, in validate_value warnings.warn(msg) UserWarning: The value length (22) exceeds the maximum length of 16 allowed for VR SH. 230412-10:38:53,694 nipype.workflow INFO: [Node] Setting-up "convert" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/dcm2niixo_3eh826/convert". INFO: [Node] Setting-up "convert" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/dcm2niixo_3eh826/convert". 230412-10:38:53,698 nipype.workflow INFO: [Node] Executing "convert" INFO: [Node] Executing "convert" 230412-10:38:53,794 nipype.interface INFO: stdout 2023-04-12T10:38:53.793922:Chris Rorden's dcm2niiX version v1.0.20220720 Clang13.0.0 x86-64 (64-bit MacOS) INFO: stdout 2023-04-12T10:38:53.793922:Chris Rorden's dcm2niiX version v1.0.20220720 Clang13.0.0 x86-64 (64-bit MacOS) 230412-10:38:53,794 nipype.interface INFO: stdout 2023-04-12T10:38:53.793922:Found 3 DICOM file(s) 230412-10:38:53,794 nipype.interface INFO: stdout 2023-04-12T10:38:53.793922:Convert 3 DICOM as /private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/pytest-of-jwodder/pytest-256/test_b0dwi_for_fmap0/sub-b0dwiForFmap/dwi/sub-b0dwiForFmap_acq-b0dwi_dwi_heudiconv106 (150x152x81x3) INFO: stdout 2023-04-12T10:38:53.793922:Found 3 DICOM file(s) INFO: stdout 2023-04-12T10:38:53.793922:Convert 3 DICOM as /private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/pytest-of-jwodder/pytest-256/test_b0dwi_for_fmap0/sub-b0dwiForFmap/dwi/sub-b0dwiForFmap_acq-b0dwi_dwi_heudiconv106 (150x152x81x3) 230412-10:38:54,437 nipype.interface INFO: stdout 2023-04-12T10:38:54.437402:Conversion required 0.701784 seconds (0.697725 for core code). INFO: stdout 2023-04-12T10:38:54.437402:Conversion required 0.701784 seconds (0.697725 for core code). 230412-10:38:54,473 nipype.workflow INFO: [Node] Finished "convert", elapsed time 0.773722s. INFO: [Node] Finished "convert", elapsed time 0.773722s. 230412-10:38:54,487 nipype.workflow INFO: [Node] Setting-up "embedder" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmetaw3s_wq60/embedder". INFO: [Node] Setting-up "embedder" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmetaw3s_wq60/embedder". 230412-10:38:54,492 nipype.workflow INFO: [Node] Executing "embedder" INFO: [Node] Executing "embedder" 230412-10:38:54,509 nipype.workflow INFO: [Node] Finished "embedder", elapsed time 0.016025s. INFO: [Node] Finished "embedder", elapsed time 0.016025s. WARNING: Storing result file without outputs 230412-10:38:54,509 nipype.workflow WARNING: Storing result file without outputs 230412-10:38:54,511 nipype.workflow WARNING: [Node] Error on "embedder" (/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmetaw3s_wq60/embedder) WARNING: [Node] Error on "embedder" (/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmetaw3s_wq60/embedder) ERROR: Embedding failed: Exception raised while executing Node embedder. Traceback: Traceback (most recent call last): File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/tag.py", line 28, in tag_in_exception yield File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 2389, in walk data_element = self[tag] ~~~~^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 939, in __getitem__ self[tag] = DataElement_from_raw(elem, character_set, self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataelem.py", line 859, in DataElement_from_raw value = convert_value(vr, raw, encoding) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 768, in convert_value return converter(byte_string, encodings, VR) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in convert_text as_strings = [convert_single_string(value, encodings, vr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in as_strings = [convert_single_string(value, encodings, vr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 557, in convert_single_string validate_value( File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/valuerep.py", line 290, in validate_value warnings.warn(msg) UserWarning: The value length (22) exceeds the maximum length of 16 allowed for VR SH. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run runtime = self._run_interface(runtime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface out = function_handle(**args) ^^^^^^^^^^^^^^^^^^^^^^^ File "", line 30, in embed_dicom_and_nifti_metadata File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/dcmstack/dcmstack.py", line 1156, in parse_and_stack results = parse_and_group(src_paths, ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/dcmstack/dcmstack.py", line 1060, in parse_and_group meta = extractor(dcm) ^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/dcmstack/extract.py", line 455, in __call__ dcm.decode() File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 551, in decode self.walk(decode_callback, recursive=False) File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 2388, in walk with tag_in_exception(tag): File "/usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 155, in __exit__ self.gen.throw(typ, value, traceback) File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/tag.py", line 32, in tag_in_exception raise type(exc)(msg) from exc UserWarning: With tag (0051, 100e) got exception: The value length (22) exceeds the maximum length of 16 allowed for VR SH. Traceback (most recent call last): File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/tag.py", line 28, in tag_in_exception yield File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 2389, in walk data_element = self[tag] ~~~~^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 939, in __getitem__ self[tag] = DataElement_from_raw(elem, character_set, self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataelem.py", line 859, in DataElement_from_raw value = convert_value(vr, raw, encoding) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 768, in convert_value return converter(byte_string, encodings, VR) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in convert_text as_strings = [convert_single_string(value, encodings, vr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in as_strings = [convert_single_string(value, encodings, vr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 557, in convert_single_string validate_value( File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/valuerep.py", line 290, in validate_value warnings.warn(msg) UserWarning: The value length (22) exceeds the maximum length of 16 allowed for VR SH. PASSED ====================== 1 passed, 119 deselected in 3.47s ======================= py3: OK (13.04=setup[6.54]+cmd[6.50] seconds) congratulations :) (13.10 seconds) ```
yarikoptic commented 1 year ago

just to make sure -- we have two issues

correct?

jwodder commented 1 year ago

@yarikoptic The first item is correct. As for the second, pytest does convert the warning into an exception, but that exception is then caught by this code. I think the more pertinent issue is that the test does not fail.

yarikoptic commented 1 year ago

Ah, thank you for digesting it for me! We indeed just allow for embedder to fail and issue a warning in normal operations. We might disable that during testing to thus allow such exceptions to bubble up.