search

HEXRD / hexrd

A cross-platform, open-source library for the analysis of X-ray diffraction data.
Other
56 stars 25 forks source link

How to convert .ge3 ff HEDM data (APS) to H5/HDF5 files which can be imported by hexrd #178

Closed minety closed 2 years ago

minety commented 3 years ago

Hi,

We just finished some in situ ff HEDM experiments at APS sector-1.

But having problems of import ff data, I was using the instruction&scripts created by CHESS where they are using .h5 data format as input.

It would be very appreciated that you could help me out.

Thanks,

Tian

joelvbernier commented 3 years ago

There are example archiving scripts on Orthros from our recent PUP run here: /clhome/sec1afpup/AFRL/PUP_AFRL_Feb21/run_archiving.py

As far as importing the ge files, there are two options:

  1. The GUI can load them directly and you can export an HDF5 or npz style imageseries (good for first look, initial state, but not for batch)
  2. using a batch script like my example, which makes use of the generic image-files adapter for imageseries (see docs here)

I would recommend converting FF data from ge3 locally on Orthros or Telemann to npz (frame-cache) imageseries, as the compression ratio is 100-1000x. You can run the GUI locally loading the npz files, or via NoMachine on Telemann if you have access.

joelvbernier commented 3 years ago

Here is an example of the YAML file that can be used to load an imageseries for ge3:

image-files:
  directory: /s1c/PUP_AFRL_feb21/ge3
  files: "lshr_r4_state0_box_ff_000372.ge3"

options:
  empty-frames: 1  # skip this many frames at the beginning of each get file
  max-frames: 0  # 0: read all; otherwise a max number of frames to read from the files
joelvbernier commented 3 years ago

Attached are the files used for archiving data at 1-ID. There are specific paths that must be modified for each experiment. The fastpar log file parser comes from aps.py and grabs scan parameters from the SPEC output. I can help you modify it for your run if you are having trouble.

run_archiving_APS.zip

joelvbernier commented 3 years ago

@minety -- there is a PR staged that will greatly simplify the import of raw images to imageseries, and batch export as well. Keep on the lookout for the 0.8.2 release (hopefully tomorrow).

minety commented 3 years ago

@joelvbernier Thanks, looking forward to it. Is there going to be an instruction on how to use it?

minety commented 3 years ago

I got the following error when I ran the run_archiving.py with our HPC. What could be the reason?

RemoteTraceback Traceback (most recent call last) RemoteTraceback: """ Traceback (most recent call last): File "//lib/python3.8/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, kwds)) File "/**/.conda/envs/hexrd_37/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar return list(map(*args)) File "//aps.py", line 44, in process_raw_mp raw = imageseries.open(yml_ims, 'image-files') File "/**/.conda/envs/hexrd_37/lib/python3.8/site-packages/hexrd/imageseries/init.py", line 18, in open adapter = reg[format](filename, kwargs) File "//.conda/envs/hexrd_37/lib/python3.8/site-packages/hexrd/imageseries/load/imagefiles.py", line 38, in init self._process_files() File "//.conda/envs/hexrd_37/lib/python3.8/site-packages/hexrd/imageseries/load/imagefiles.py", line 113, in _process_files info = FileInfo(imgf, kw) File "/**/.conda/envs/hexrd_37/lib/python3.8/site-packages/hexrd/imageseries/load/imagefiles.py", line 210, in init img = fabio.open(filename) File "//.conda/envs/hexrd_37/lib/python3.8/site-packages/fabio/openimage.py", line 165, in openimage obj = _openimage(filename) File "//.conda/envs/hexrd_37/lib/python3.8/site-packages/fabio/openimage.py", line 239, in _openimage raise IOError("Fabio could not identify " + filename) OSError: Fabio could not identify //ge3/_s0_ff_000715.ge3 """

The above exception was the direct cause of the following exception:

OSError Traceback (most recent call last)

in 118 print(yml_names) 119 with multiprocessing.Pool(**kwargs) as pool: --> 120 result = pool.map(aps.process_raw_mp, yml_names) 121 122 ~/.conda/envs/hexrd_37/lib/python3.8/multiprocessing/pool.py in map(self, func, iterable, chunksize) 362 in a list that is returned. 363 ''' --> 364 return self._map_async(func, iterable, mapstar, chunksize).get() 365 366 def starmap(self, func, iterable, chunksize=None): ~/.conda/envs/hexrd_37/lib/python3.8/multiprocessing/pool.py in get(self, timeout) 769 return self._value 770 else: --> 771 raise self._value 772 773 def _set(self, i, obj): OSError: Fabio could not identify /****/ge3/****_s0_ff_000715.ge3 Thanks
joelvbernier commented 3 years ago

@minety it looks like you may have the wrong version of Fabio installed? Can you do the following:

conda list fabio

and share the result with me?

joelvbernier commented 3 years ago

Also could be a problem with the path; can you tell me the absolute path to the images and share one of the raw_*.yml files?

minety commented 3 years ago

Thanks for your suggestion, it is the fabio version problem (I had a 0.10 version before).

joelvbernier commented 2 years ago

I will be adding archiving examples for APS and CHESS to the examples repo soon.