Loading yamls in gui /hexrd

darrencpagan commented 10 months ago

I am having a problem loading instrument (configuration) .ymls in the new (1.0.0) gui release. I am not having the same issue loading .h5 / .hexrd configurations of the same detector scheme. Error below.


Traceback (most recent call last):
  File "/nfs/chess/user/dcp99/sw/miniconda/envs/20231029_hexrdgui/lib/python3.11/site-packages/hexrdgui/main_window.py", line 417, in on_action_open_config_file_triggered
    HexrdConfig().load_instrument_config(str(path))
  File "/nfs/chess/user/dcp99/sw/miniconda/envs/20231029_hexrdgui/lib/python3.11/site-packages/hexrdgui/hexrd_config.py", line 994, in load_instrument_config
    self.config['instrument'] = formats[ext]()
                                ^^^^^^^^^^^^^^
  File "/nfs/chess/user/dcp99/sw/miniconda/envs/20231029_hexrdgui/lib/python3.11/site-packages/hexrdgui/hexrd_config.py", line 964, in read_yaml
    conf = yaml.load(f, Loader=NumPyIncludeLoader)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nfs/chess/user/dcp99/sw/miniconda/envs/20231029_hexrdgui/lib/python3.11/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nfs/chess/user/dcp99/sw/miniconda/envs/20231029_hexrdgui/lib/python3.11/site-packages/yaml/constructor.py", line 51, in get_single_data
    return self.construct_document(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nfs/chess/user/dcp99/sw/miniconda/envs/20231029_hexrdgui/lib/python3.11/site-packages/yaml/constructor.py", line 60, in construct_document
    for dummy in generator:
  File "/nfs/chess/user/dcp99/sw/miniconda/envs/20231029_hexrdgui/lib/python3.11/site-packages/yaml/constructor.py", line 413, in construct_yaml_map
    value = self.construct_mapping(node)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nfs/chess/user/dcp99/sw/miniconda/envs/20231029_hexrdgui/lib/python3.11/site-packages/yaml/constructor.py", line 218, in construct_mapping
    return super().construct_mapping(node, deep=deep)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nfs/chess/user/dcp99/sw/miniconda/envs/20231029_hexrdgui/lib/python3.11/site-packages/yaml/constructor.py", line 143, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nfs/chess/user/dcp99/sw/miniconda/envs/20231029_hexrdgui/lib/python3.11/site-packages/yaml/constructor.py", line 100, in construct_object
    data = constructor(self, node)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/nfs/chess/user/dcp99/sw/miniconda/envs/20231029_hexrdgui/lib/python3.11/site-packages/yaml/constructor.py", line 427, in construct_undefined
    raise ConstructorError(None, None,
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:numpy.core.multiarray.scalar'

darrencpagan commented 10 months ago

This appears to be more universal and I get the same errors when trying to load yamls through HEXRD core functionalities in scripts.

joelvbernier commented 10 months ago

looks like it might be getting confused converting some field to array data? Sent from my iPhoneOn Nov 28, 2023, at 08:42, darrencpagan @.***> wrote: This appears to be more universal and I get the same errors when trying to load yamls through HEXRD core functionalities in scripts.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

psavery commented 10 months ago

It looks like there were numpy scalars written out to the yaml file. Can you send me a copy of a yaml file like this so I can look at it?

darrencpagan commented 10 months ago

You are right. My apologies, I hadn't looked at the file itself. This yaml file was dumped directly out the GUI.

beam:
  energy: !!python/object/apply:numpy.core.multiarray.scalar
  - &id001 !!python/object/apply:numpy.dtype
    args:
    - f8
    - false
    - true
    state: !!python/tuple
    - 3
    - <
    - null
    - null
    - null
    - -1
    - -1
    - 0
  - !!binary |
    AiuHFtn+VEA=
  source_distance: !!python/object/apply:numpy.core.multiarray.scalar
  - *id001
  - !!binary |
    AAAAAAAA8H8=
  vector:
    azimuth: 90.0
    polar_angle: 90.0
detectors:
  ff1:
    buffer:
    - 2.0
    - 2.0
    detector_type: planar
    pixels:
      columns: 3072
      roi: null
      rows: 3888
      size:
      - 0.0748
      - 0.0748
    saturation_level: 15800.0
    transform:
      tilt:
      - 0.0
      - 0.0
      - 0.0
      translation:
      - 111.0417371476068
      - 2.2833527798010738
      - -1221.8177356141002
  ff2:
    buffer:
    - 2.0
    - 2.0
    detector_type: planar
    pixels:
      columns: 3072
      roi: null
      rows: 3888
      size:
      - 0.0748
      - 0.0748
    saturation_level: 15800.0
    transform:
      tilt:
      - 0.0
      - 0.0
      - 0.0
      translation:
      - -130.3849984715735
      - 2.2452296350368535
      - -1221.2267270720567
id: dexelas_id3a_20200130
oscillation_stage:
  chi: !!python/object/apply:numpy.core.multiarray.scalar
  - *id001
  - !!binary |
    AAAAAAAAAAA=
  translation:
  - 0.0
  - 0.0
  - 0.0

psavery commented 10 months ago

You are right. My apologies, I hadn't looked at the file itself. This yaml file was dumped directly out the GUI.

Yes, I think we need to fix an issue with the yaml writing.

psavery commented 10 months ago

@darrencpagan Here is that same file with the numpy types converted to basic types:

beam:
  energy: 83.982
  source_distance: .inf
  vector:
    azimuth: 90.0
    polar_angle: 90.0
detectors:
  ff1:
    buffer:
    - 2.0
    - 2.0
    detector_type: planar
    pixels:
      columns: 3072
      roi: null
      rows: 3888
      size:
      - 0.0748
      - 0.0748
    saturation_level: 15800.0
    transform:
      tilt:
      - 0.0
      - 0.0
      - 0.0
      translation:
      - 111.0417371476068
      - 2.2833527798010738
      - -1221.8177356141002
  ff2:
    buffer:
    - 2.0
    - 2.0
    detector_type: planar
    pixels:
      columns: 3072
      roi: null
      rows: 3888
      size:
      - 0.0748
      - 0.0748
    saturation_level: 15800.0
    transform:
      tilt:
      - 0.0
      - 0.0
      - 0.0
      translation:
      - -130.3849984715735
      - 2.2452296350368535
      - -1221.2267270720567
id: dexelas_id3a_20200130
oscillation_stage:
  chi: 0.0
  translation:
  - 0.0
  - 0.0
  - 0.0

You should be able to load this one in.

PR HEXRD/hexrd#588 ensures that we don't dump numpy types, but convert them to native. Once that is merged, you hopefully shouldn't see this issue again.

If you have other yaml files with numpy types inside them, then after that PR is merged and you update your HEXRD, you can convert them through code similar to the following:

import yaml

from hexrd.utils.yaml import NumpyToNativeDumper

with open('unsafe.yml', 'r') as rf:
    conf = yaml.unsafe_load(rf)

with open('safe.yml', 'w') as wf: 
    yaml.dump(conf, wf, Dumper=NumpyToNativeDumper)

HEXRD / hexrdgui

Loading yamls in gui /hexrd #1625