ERROR: fitgrain error after CeO2 calibration and gold indexing

[minety]

I did the CeO2 calibration and gold indexing successfully. But when I try to do fitgrain, I got an error as attached figures.

Could you please tell me what's the problem?

Thanks,

Tian

[joelvbernier]

There are a few things that could lead to no reflections being simulated, but it is usually either the omega information is incorect, or the HKLs selected is too restrictive. Can you export an npz file for your imageseries and attach it here along with an instrument config and I'll take a look?

[minety]

Au_FF_data.zip

@joelvbernier Thanks a lot, please find attached for the .npz file and instrument config.

Regarding the config for indexing and grain fitting, I also have several questions as follows:

1. Indexing config:
   What's the meaning of _clustering>radius, eta>mask, seed_search>method>label>filter_radius_

2. Fit grains config: What's the meaning of npdiv? What's the difference between tth_max>true and tth_max>false?

Best, Tian

[joelvbernier]

Ah, indeed your omega information is the issue here:

Unfortunately, this critical metadata is not attached to the data automatically at the beamline. I have APS-specific scripts to use the scan logs (fastpar files) to automatically archive the scans based on sample name, but when you load the imageseries into the GUI from raw GE files, you must input the omega metadata by hand. You were teling it that your scan was only over 0.25 degrees, so for a single grain it isn't surprising that no reflections were found in that very small slice.

As far as I know, for APS 1-ID, your values should be:

• empty frames: 1
• start ω: 0.
• stop ω: 360.
• steps (this has replaced entering δω): 1440 (this will automatically update with the empty frames set to 1)

Also -- as I am not sure whether or not you already accounted for this -- the APS GE3 images are "upside-down" and need to be mirrored about the horizontal to put them into the proper configuration in the software. Did you do this when you made the npz file?

[joelvbernier]

Fixed your metadata and trimmed the imageseries to ignore the first frame and was able to index/fit:

It is what I expect, a single grain near the origin, and a pair of twins ~500µm away in X (at ω=0).

[joelvbernier]

I did notice that we need to add the option to 'trim' frames from npz and hdf5 style imageseries (which are handled a little differently internally). Draft pull request coming...

In the meantime, I 'corrected' your imageseries interactively in IPython via the following:

In [47]: import numpy as np

In [48]: from hexrd import imageseries

In [49]: ims = imageseries.open("FF_Au_10_detector_1.npz", format='frame-cache')

In [50]: len(ims)
Out[50]: 1441

In [51]: pims = imageseries.process.ProcessedImageSeries(ims, [], frame_list=np.arange(1, 1441))

In [52]: len(pims)
Out[52]: 1440

In [53]: pims.metadata['omega']
Out[53]: 
array([[0.00000000e+00, 1.73490632e-04],
       [1.73490632e-04, 3.46981263e-04],
       [3.46981263e-04, 5.20471895e-04],
       ...,
       [2.49479528e-01, 2.49653019e-01],
       [2.49653019e-01, 2.49826509e-01],
       [2.49826509e-01, 2.50000000e-01]])

In [54]: ome_range = 0.25*np.arange(0, 1441)

In [55]: pims.metadata['omega'] = np.vstack([ome_range[:-1], ome_range[1:]]).T

In [56]: pims.metadata['omega']
Out[56]: 
array([[0.0000e+00, 2.5000e-01],
       [2.5000e-01, 5.0000e-01],
       [5.0000e-01, 7.5000e-01],
       ...,
       [3.5925e+02, 3.5950e+02],
       [3.5950e+02, 3.5975e+02],
       [3.5975e+02, 3.6000e+02]])

In [57]: imageseries.write(pims, fname='dummy', fmt='frame-cache', threshold=10, cache_file='test.npz')

In [58]: 

[joelvbernier]

The aforementioned option is being handled via #910

[minety]

Hi Joel,

@joelvbernier Thanks a lot, I got the same result as yours for gold calibrant, however, for my stainless steel sample (the .npz and config files are attached in the dropbox link below) I cannot find any grains, and hexrd was always killed by the system after about an hour of finding. (Our system has 20 CPUs and 250 GB memory)

I check the eta-omega maps for different hkls and found the peaks were very crowded. Do you have any ideas to filter and process the data so that they can be correctly indexed?

https://www.dropbox.com/s/zhfjn9e187hbhl8/FF_sam3_s0_layer1.zip?dl=0

Best, Tian

[joelvbernier]

You need to be careful about the thresholds; I can take a look at your npz

Sent from my iPhone

On May 6, 2021, at 23:11, Ye Tian @.***> wrote:

 Hi Joel,

@joelvbernier Thanks a lot, I got the same result as yours for gold calibrant, however, for my stainless steel sample (the .npz and config files are attached in the dropbox link below) I cannot find any grains, and hexrd was always killed by the system after about an hour of finding. (Our system has 20 CPUs and 250 GB memory)

I check the eta-omega maps for different hkls and found the peaks were very crowded. Do you have any ideas to filter and process the data so that they can be correctly indexed?

https://www.dropbox.com/s/zhfjn9e187hbhl8/FF_sam3_s0_layer1.zip?dl=0

Best, Tian

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[minety]

@joelvbernier Thanks, my npz was in the dropbox link I attached. I tried different thresholds but all ended up with nothing but got killed by the system/software. My yaml files are also attached.

Best, Tian

[joelvbernier]

@minety -- I have your test problem. I looks like the austenite has a lot of grains, so just a matter of adjusting the indexing parameters. For very large number of grains, it is typically best to run on a workstation or HPC node with a decent number of CPUs to speed things up. We are working on MPI parallel versions of fitting as well.

[minety]

analysis_name: sam3_ff_s0_fitgrains_fr4_cr0p3_0p92_50 # defaults to analysis

working_dir: /home/ytian37/scratch/2021Feb_RAMS/FF_analysis/sam3_s0/

multiprocessing: all # "all", or "half", or -1 means all but one, defaults to -1

material:
material:
definitions: RAMS_Materials.h5
active: Fe-austenite
dmin: 1 # defaults to 1.0 angstrom
tth_width: 0.25 # defaults to 0.25 degrees
min_sfac_ratio: 0.05 # min percentage of max |F|^2 to exclude; default None

image_series:
format: frame-cache
data:

• file: hufnagel_sam3_s0_ff_000625_ge3.npz
  args: {}
  panel: GE # must match detector key

instrument: ge_detector.yml

find_orientations:
orientation_maps:

A file name must be specified. If it doesn't exist, one will be created

file: null                                                                  

threshold: 50  #find_orientations                                           
bin_frames: 1 # defaults to 1                                                             
active_hkls: [0,1,2,3]                                                      

seed_search: # this section is ignored if use_quaternion_grid is defined
hkl_seeds: [0,1,2,3] # hkls ids to use, must be defined for seeded search
fiber_step: 0.5 # degrees, defaults to ome tolerance

method:                                                                     
  label:                                                                    
    filter_radius: 4 
    threshold: 50 # defaults to 1                                          

threshold: 50 #scoring?

omega:
tolerance: 0.5 # in degrees, defaults to 2x ome step

eta:
tolerance: 1 # in degrees, defaults to 2x ome step
mask: 5 # degrees, mask angles close to ome rotation axis, defaults to 5

clustering:
radius: 0.3 # misorientation search range
completeness: 0.92 # completeness threshold
algorithm: dbscan

analysis_name: sam3_ff_s0_fitgrains_fr4_cr0p3_0p92_50 # defaults to analysis

image_series:
format: frame-cache
data:

• file: hufnagel_sam3_s0_ff_000625_ge3.npz
  args: {}
  panel: GE # must match detector key

fit_grains: do_fit: true # if false, extracts grains but doesn't fit. defaults to true

estimate: sam3_ff_s0_findori_fr4_cr0p3_0p92_50/grains.out

npdiv: 4 # number of polar pixel grid subdivisions, defaults to 2

threshold: 1

active_hkls: [2,3]

tolerance: tth: [0.25, 0.20, 0.15, 0.1, 0.05, 0.05, 0.03] # tolerance lists must be identical length eta: [1.0, 1.0, 0.5, 0.3, 0.2, 0.1, 0.05] omega: [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]

refit: [2, 1.1]

tth_max: 9.344 # true, false, or a non-negative value, defaults to true

The above script is the yaml file for findorientations and fitgrains. I did find the right parameters for indexing annealed state using HPC, however, I got some problems finding most of the grains for the ### 4% engineering strain step (I only found about 1/4 of the grains).

Which parameter can I adjust to find more deformed grains?

My thoughts are to adjust ### threshold to separate overlapping peaks due to the large misorientation spread of individual grains.

What about tweaking the filtered radius and clustering radius? Will these help with finding more deformed grains?

Thanks a lot,

Tian

@joelvbernier

[joelvbernier]

You can look at what peaks are getting labeled in the seeded search (findorientations) in the GUI; this is in the eta-omega widget: A word of caution, however: the size of the scored orientation list above the min_completeness threshold shouldn't exceed ~1e7, otherwise it might overwhelm dbscan. It is in general very difficult to get every grain in very densely populated volumes due to dynamic range limitations. You will almost always miss the smallest grains in the first pass. However, we can take a second pass with the signals from the previously indexed grains removed if you really want to try to fit the smaller grains.

There are also multiple options to find peaks for the seeded search. In the case of the austenite, labeling might work better to find more peaks. I do find that filtering the maps (top left corner) seems necessary to separate the areas with a high density of peaks. Play with it a bit and you'll get a feel for what the different parameters are doing.

[minety]

@joelvbernier Thanks for your suggestions. In terms of the second pass, how can I remove signals from the previously indexed grains? How can I do that with .yml file? Tian

[joelvbernier]

@minety -- we will have to build a new tool for this.