Previously working EDD map pipeline now fails

[arwoll]

Describe the bug A previously working EDD map pipeline constructed in December now fails to work

To Reproduce Provide a minimal pipeline configuration in which the bug appears:

config:
  root: /nfs/chess/auxiliary/cycles/2023-3/id1a3/ko-3538-b/
  inputdir: raw_data/edd23-char-1/
  outputdir: reduced_data/analysis_ARW/
  log_level: debug

pipeline:
  - common.MapReader:
      map_config:
        title: edd-char-1-87
        station: id1a3
        experiment_type: EDD
        sample:
          name: ceria
        spec_scans:
        - spec_file: spec.log
          scan_numbers: 
          - 87
        independent_dimensions:
        - label: samy
          units: mm
          data_type: spec_motor
          name: sampYcp
        # ADD ANOTHER IND. DIM. IF NEEDED
      detector_names: [0,2,3,5,6,7,8,10,13,14,16,17,18,19,21,22]
  - common.NexusWriter:
      filename: edd-char-1-87-TEST.nxs
      force_overwrite: true

Expected behavior Should produce the nexus file listed above, edd-char-1-87-TEST.nxs.

Screenshots (CHAP_edd) [aw30@lnx-id3b-2 analysis_ARW]$ CHAP edd-char-1-87-ceria-map-pipeline.yaml CHAP.runner : INFO: Input pipeline configuration: [{'common.MapReader': {'map_config': {'title': 'edd-char-1-87', 'station': 'id1a3', 'experiment_type': 'EDD', 'sample': {'name': 'ceria'}, 'spec_scans': [{'spec_file': 'spec.log', 'scan_numbers': [87]}], 'independent_dimensions': [{'label': 'samy', 'units': 'mm', 'data_type': 'spec_motor', 'name': 'sampYcp'}]}, 'detector_names': [0, 2, 3, 5, 6, 7, 8, 10, 13, 14, 16, 17, 18, 19, 21, 22]}}, {'common.NexusWriter': {'filename': 'edd-char-1-87-TEST.nxs', 'force_overwrite': True}}]

CHAP.runner : INFO: Loaded <CHAP.common.reader.MapReader object at 0x7fdb7a015510> CHAP.runner : INFO: Loaded <CHAP.common.writer.NexusWriter object at 0x7fdb7a0156f0> CHAP.runner : INFO: Loaded <CHAP.pipeline.Pipeline object at 0x7fdb7a0153c0> with 2 items

CHAP.runner : INFO: Calling "execute" on <CHAP.pipeline.Pipeline object at 0x7fdb7a0153c0> Pipeline : INFO: Executing "execute"

Pipeline : INFO: Calling "execute" on <CHAP.common.reader.MapReader object at 0x7fdb7a015510> MapReader : DEBUG: Executing "read" with {'inputdir': '/nfs/chess/previousid1a3/2023-3/ko-3538-b/edd23-char-1', 'map_config': {'title': 'edd-char-1-87', 'station': 'id1a3', 'experiment_type': 'EDD', 'sample': {'name': 'ceria'}, 'spec_scans': [{'spec_file': 'spec.log', 'scan_numbers': [87]}], 'independent_dimensions': [{'label': 'samy', 'units': 'mm', 'data_type': 'spec_motor', 'name': 'sampYcp'}]}, 'detector_names': [0, 2, 3, 5, 6, 7, 8, 10, 13, 14, 16, 17, 18, 19, 21, 22]} MapReader : INFO: Executing "read" Traceback (most recent call last): File "/nfs/chess/user/aw30/Git_Repos/ChessAnalysisPipeline/CHAP/common/models/map.py", line 308, in validate_for_spec_scans self.get_value(scans, scan_number, index) File "/nfs/chess/user/aw30/Git_Repos/ChessAnalysisPipeline/CHAP/common/models/map.py", line 387, in get_value return get_spec_motor_value(spec_scans.spec_file, File "/nfs/chess/user/aw30/Git_Repos/ChessAnalysisPipeline/CHAP/common/models/map.py", line 446, in get_spec_motor_value scanparser.get_spec_scan_motor_vals( File "/nfs/chess/user/aw30/Git_Repos/ChessAnalysisPipeline/CHAP/utils/scanparsers.py", line 1213, in get_spec_scan_motor_vals raise NotImplementedError('Only relative motor values are available.') NotImplementedError: Only relative motor values are available.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/nfs/chess/user/aw30/miniconda3/envs/CHAP_edd/bin/CHAP", line 33, in sys.exit(load_entry_point('ChessAnalysisPipeline', 'console_scripts', 'CHAP')()) File "/nfs/chess/user/aw30/Git_Repos/ChessAnalysisPipeline/CHAP/runner.py", line 100, in main runner(run_config, pipeline_config) File "/nfs/chess/user/aw30/Git_Repos/ChessAnalysisPipeline/CHAP/runner.py", line 115, in runner run(pipeline_config, File "/nfs/chess/user/aw30/Git_Repos/ChessAnalysisPipeline/CHAP/runner.py", line 218, in run pipeline.execute() File "/nfs/chess/user/aw30/Git_Repos/ChessAnalysisPipeline/CHAP/pipeline.py", line 44, in execute data = item.execute(data=data, kwargs) File "/nfs/chess/user/aw30/Git_Repos/ChessAnalysisPipeline/CHAP/pipeline.py", line 168, in execute data = method(args) File "/nfs/chess/user/aw30/Git_Repos/ChessAnalysisPipeline/CHAP/common/reader.py", line 121, in read map_config = MapConfig(**map_config, inputdir=inputdir) File "pydantic/main.py", line 339, in pydantic.main.BaseModel.init File "pydantic/main.py", line 1076, in pydantic.main.validate_model File "pydantic/fields.py", line 895, in pydantic.fields.ModelField.validate File "pydantic/fields.py", line 928, in pydantic.fields.ModelField._validate_sequence_like File "pydantic/fields.py", line 1094, in pydantic.fields.ModelField._validate_singleton File "pydantic/fields.py", line 898, in pydantic.fields.ModelField.validate File "pydantic/fields.py", line 1151, in pydantic.fields.ModelField._apply_validators File "pydantic/class_validators.py", line 339, in pydantic.class_validators._generic_validator_basic.lambda14 File "/nfs/chess/user/aw30/Git_Repos/ChessAnalysisPipeline/CHAP/common/models/map.py", line 568, in validate_data_source_for_map_config return _validate_data_source_for_map_config(data_source, values) File "/nfs/chess/user/aw30/Git_Repos/ChessAnalysisPipeline/CHAP/common/models/map.py", line 566, in _validate_data_source_for_map_config data_source.validate_for_spec_scans(values.get('spec_scans')) File "/nfs/chess/user/aw30/Git_Repos/ChessAnalysisPipeline/CHAP/common/models/map.py", line 310, in validate_for_spec_scans raise RuntimeError( RuntimeError: Could not find data for sampYcp (data_type "spec_motor") on scan number 87 for index 0 in spec file /nfs/chess/previousid1a3/2023-3/ko-3538-b/edd23-char-1/spec.log

Environment (please complete the following information): This was run on the cluster using the current version of the edd-spring2024 branch.

Additional context I briefly thought the path was not properly resolving (my pipeline specifies

root: /nfs/chess/auxiliary/cycles/2023-3/id1a3/ko-3538-b/ 
inputdir: raw_data/edd23-char-1/

whereas the Traceback complains about the location:

/nfs/chess/previousid1a3/2023-3/ko-3538-b/edd23-char-1/spec.log

but these are clearly resolving to the same file.

Based on the error message "Only relative motor values are available" I suspect that this error is caused by changes made to accommodate the current EDD workflow.

[keara-soloway]

I just pushed an update to the treatment of spec_motors for EDD maps that should fix the "Only relative motor values are available" issue. However, there's still a problem with this version: a .par file is now required to get motor postions, but there is no .par file for the sample directory you're looking at. Is it possible that the .par & .json files got lost in the move to previousid1a3, or did this sample never have those files at all?

[keara-soloway]

Fixed in 675e060822695fc7d7a62d1dc6a5d4aeecf1e628.