EMEBSD, version:-5.0.20200909.-Win64, The simulated patterns vary largely with experimental patterns

[luqi-cloud]

I want to perform CCEBSD with the simulated patterns via openXY. But the simulated patterns are very different from the experimental patterns. Are these differences caused by deformation tensor?

Actually, I don't know the exact CCD pixel size. I use a default value 60.

The pattern center optimized by Aztec is : Pcx = 0.48, Pcy=0.548, z = 0.508 The ctf and cpr files have also been included in the file named 'attachment.zip' attachment.zip

Experimental: Simulation:

&EBSDdata ! template file for the EMEBSD program ! ! distance between scintillator and illumination point [microns] L = 15384.96, ! tilt angle of the camera (positive below horizontal, [degrees]) thetac = 10.0, ! CCD pixel size on the scintillator surface [microns] delta = 60.0, ! number of CCD pixels along x and y numsx = 622, numsy = 512, ! pattern center coordinates in units of pixels xpc = -12.44, ypc = 24.576, ! angle between normal of sample and detector omega = 0.0, ! transfer lens barrel distortion parameter alphaBD = 0.0, ! energy range in the intensity summation [keV] energymin = 10.0, energymax = 20.0, ! include a realistic intensity background or not ... includebackground = 'n', ! name of angle file (euler angles or quaternions); path relative to EMdatapathname anglefile = 'euler.txt', ! does this file have only orientations ('orientations') or does it also have pattern center and deformation tensor ('orpcdef') ! if anglefiletype = 'orpcdef' then each line in the euler input file should look like this: (i.e., 15 floats) ! 55.551210 58.856774 325.551210 0.0 0.0 15000.0 1.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 1.00 ! <- Euler angles (degrees) -> <- pat. ctr. -> <- deformation tensor in column-major form-> ! anglefiletype = 'orientations', ! 'tsl' or 'hkl' Euler angle convention parameter eulerconvention = 'hkl', ! name of EBSD master output file; path relative to EMdatapathname masterfile = 'Al_master_20kV.h5', energyfile = 'Al_master_20kV.h5', ! name of output file; path relative to EMdatapathname datafile = '11-04-2.h5', ! bitdepth '8bit' for [0..255] bytes; 'float' for 32-bit reals; '##int' for 32-bit integers with ##-bit dynamic range ! e.g., '9int' will get you 32-bit integers with intensities scaled to the range [ 0 .. 2^(9)-1 ]; ! '17int' results in the intensity range [ 0 .. 2^(17)-1 ] bitdepth = '8bit', ! incident beam current [nA] beamcurrent = 150.0, ! beam dwell time [micro s] dwelltime = 100.0, ! include Poisson noise ? (y/n) (noise will be applied before binning and intensity scaling) poisson = 'n', ! binning mode (1, 2, 4, or 8) binning = 1, ! should we perform an approximate computation that includes a lattice distortion? ('y' or 'n') ! This uses a polar decomposition of the deformation tensor Fmatrix which results in ! an approcimation of the pattern for the distorted lattice; the bands will be very close ! to the correct position in each pattern, but the band widths will likely be incorrect. applyDeformation = 'n' ! if applyDeformation='y' then enter the 3x3 deformation tensor in column-major form ! the default is the identity tensor, i.e., no deformation Ftensor = 1.D0, 0.D0, 0.D0, 0.D0, 1.D0, 0.D0, 0.D0, 0.D0, 1.D0, ! intensity scaling mode 'not' = no scaling, 'lin' = linear, 'gam' = gamma correction scalingmode = 'gam', ! gamma correction factor gammavalue = 2.5, !======================= ! if the 'makedictionary' parameter is 'n', then we have the normal execution of the program ! if set to 'y', then all patterns are pre-processed using the other parameters below, so that ! the resulting dictionary can be used for static indexing in the EMEBSDDI program... ! these parameters must be taken identical to the ones in the EMEBSDDI.nml input file to have ! optimal indexing... makedictionary = 'n', ! should a circular mask be applied to the data? 'y', 'n' maskpattern = 'n', ! mask radius (in pixels, AFTER application of the binning operation) maskradius = 240, ! hi pass filter w parameter; 0.05 is a reasonable value hipassw = 0.05, ! number of regions for adaptive histogram equalization nregions = 10, !======================= ! number of threads (default = 1) nthreads = 10, /

[hakonanes]

Why do you use a gamma value of 2.5? It might be that this is the reason for the dark simulated pattern. A value of 1/3 is recommended in the dictionary indexing tutorial paper (bottom of page 240): https://doi.org/10.1007/s40192-019-00137-4.

Performing a gamma contrast adjustement with gammas of 1.75 and 2.5 on your experimental pattern using scikit-image gives me this result (both intensity distributions in the range [0, 1]), indicating that a higher gamma gives in general a darker image:

Edit: Python code:

>>> import matplotlib.pyplot as plt
>>> from skimage import exposure
>>> im = plt.imread("98073457-58029980-1ea3-11eb-8ad1-cd5121fe204a.png")
>>> im_gamma_corrected_2_5 = exposure.adjust_gamma(im, 2.5)
>>> im_gamma_corrected_1_75 = exposure.adjust_gamma(im, 1.75)
>>> fig, ax = plt.subplots(figsize=(8, 6), ncols=3)
>>> ax[0].imshow(im, cmap="gray")
>>> ax[0].set_title("Original")
>>> ax[0].axis("off")
>>> ax[1].imshow(im_gamma_corrected_1_75, cmap="gray")
>>> ax[1].set_title(r"$\gamma$ = 1.75")
>>> ax[1].axis("off")
>>> ax[2].imshow(im_gamma_corrected_2_5, cmap="gray")
>>> ax[2].set_title(r"$\gamma$ = 2.5")
>>> ax[2].axis("off")
>>> fig.savefig("gamma_correction.png", bbox_inches="tight", pad_inches=0)

[luqi-cloud]

Thanks for your reply. Yes, a value of 1/3 make patterns brighter. But the inclination angle of the Kikuchi belt is more important.

Experimental:

Simulation:

[hakonanes]

My guess is that either (1) the pattern center calculation from Oxford's to EMsoft's convention isn't correct (detector pixel size?), or (2) the pattern center is incorrect...

[luqi-cloud]

I get pattern center, piexel size from Aztec (a software provided by Oxford). I calculate the pattern based on the equations according to https://doi.org/10.1007/s40192-019-00137-4.

[elena-pascal]

Is the camera tilt also correct?

[luqi-cloud]

I try tilt angle from 0 to 10, but always get poor performance. The tilt angle is not included in the Oxford software(Aztec), ctf and cpr file. Maybe I didn't find it. I only have the following image, which tells me a small tilt angle.

[hakonanes]

The band angles look more or less correct with zero camera tilt, right?

It looks like you just have to "zoom into" the pattern, i.e. increase the specimen-to-scintillator distance L, presumably by increasing the detector pixel size?

[elena-pascal]

haha such timing! I was just looking at this, and indeed your L is slightly off, I think you might have used Ny instead Nx in that formula above. I used a value of 18958.56 and a tilt zero as @hakonanes suggested and the angles look a bit better (see below).

However, the xPC and yPC seem still off. It could be limitations of the Oxford software. EMsoft comes with EMfit if you have access to an IDL licence.

[ChaoyiZhu93]

oxford embeds all the info regarding detector geometry, kV, etc as part of the pattern metadata. The detector-orientation-euler2-deg is the angle between the electron column and detector's optical axis (0 detector tilt in this case). in addition, the pattern-center-x-pu needs to be flipped (1-pattern-center-x-pu) when you convert the HKL pc to EMsoft pc (or equivalently -xpc) due to a recent change for the view point (detector to sample). the detector pixel size is about 22.91 for the oxford symmetry1 detector.

     <pattern-center-x-pu>0.47859156295901767</pattern-center-x-pu>
     <pattern-center-y-pu>0.54362409081130969</pattern-center-y-pu>
     <detector-distance-pu>0.50848986359058135</detector-distance-pu>
     <sem-acc-voltage-kv>20</sem-acc-voltage-kv>
     <sem-working-distance-mm>16.487165451049805</sem-working-distance-mm>
     <specimen-tilt-deg>70</specimen-tilt-deg>
     <specimen-tilt-axis>X</specimen-tilt-axis>
     <detector-orientation-euler1-deg>0.88773426991828519</detector-orientation-euler1-deg>
     **<detector-orientation-euler2-deg>89.656555256406222</detector-orientation-euler2-deg>**
     <detector-orientation-euler3-deg>359.1140052486561</detector-orientation-euler3-deg>
     <lens-distortion>0</lens-distortion>
     <lens-field-of-view-mm>23.6946875</lens-field-of-view-mm>
     <detector-insertion-distance-mm>114.59760284423828</detector-insertion-distance-mm>
     <beam-position-offset-x-um>-21.571566735150025</beam-position-offset-x-um>
     <beam-position-offset-y-um>21.721566735150024</beam-position-offset-y-um>

[luqi-cloud]

Thanks for all reply. But I'm still confused about how to calculate pattern center. Pattern center after flipped: EMsoft_pcx = 1 - 0.48 - pu EMsoft_pcy = 1 - 0.54 - pu Is it right? pu = 0.5 ?

[luqi-cloud]

I tried several combinations of pcx, pcy and tilt angle. Finally, I use the following parameters. But it seems the parameters I selected are very different from those given in the literature https://doi.org/10.1007/s40192-019-00137-4.

! distance between scintillator and illumination point [microns] L = 15300.0, ! tilt angle of the camera (positive below horizontal, [degrees]) thetac = -2.0, ! CCD pixel size on the scintillator surface [microns] delta = 50.0, ! number of CCD pixels along x and y numsx = 622, numsy = 512, ! pattern center coordinates in units of pixels xpc = 20, ypc = 90,

Experimental:

Simulated:

[elena-pascal]

I'm not sure what you mean by different from literature. These are all experimental parameters, unless you work on the exact same patterns as the ones in the paper it would be strange to use the same parameters.

Maybe the not intuitive thing to flag up here is that L scales with the pixel size in the way defined in EMsoft.

[luqi-cloud]

I mean the parameters I selected are very different from the parameters calculated by equations in reference https://doi.org/10.1007/s40192-019-00137-4.

[elena-pascal]

Ok. You can indeed generate similar looking images for very different looking set of parameters. This is because:

1. Crystal symmetry
2. EMsoft parameters scale with pixel size, so you can, in a way, bin your pixels and get a new looking set of parameters, while keeping the geometry the same. I'll venture to say that this is happening here.

Also to note, the transformation given in the paper is from the HKL frame of reference used by Oxford to the reference frame used by EMsoft. @ChaoyiZhu93 rightly pointed out that since the paper came out EMsoft flipped its x-axis as described here. This in practice means:

So what you did is double your pixel size, L and x_PC.

Now, y is very off indeed, I had to go from calculated 27.1 to 74 (with 22.91 pixel size), just to be in the same area. I am having another look at the EMsoft geometry but it could well be the Oxford software giving poor estimates of pattern centre.

[ChaoyiZhu93]

@elena-pascal has correctly provided the modified expression for the Xpc in EMsoft.

Using global optimization, the pc that I found to match well was (0.477, 0.649, 0.511) (in HKL pc convention) which corresponds to (14.367, 76.033, 7288.407) (in EMsoft pc convention).

I also optimized the Euler angles to correct for any inaccuracy in the absolute orientation (this could be due to many sources of error, typically~1-2 deg). The Euler angles I found [180.92, 7.11, 58.19] which is about 1.7 deg misorientation from the original Euler angle.

[luqi-cloud]

Thanks for all answer, it works well now.

[gsparks3]

Now, y is very off indeed, I had to go from calculated 27.1 to 74 (with 22.91 pixel size), just to be in the same area. I am having another look at the EMsoft geometry but it could well be the Oxford software giving poor estimates of pattern centre.

I just finished dealing with some observed pattern / simulated pattern mismatch issues with an Oxford Symmetry detector that ended up having a very similar cause - when Marc fit the pattern center for me, the final yPC value was significantly different than what the Oxford software reported. So, it seems that yPC error may be a consistent issue with AZtec. The xPC and zPC were also off, but by much smaller amounts that seemed well within usual limits.

PC from software: x = 0.52224 y = 0.64129 z* = 0.47906

PC from fit: x = 0.52604 (+0.0038) y = 0.79064 (+0.1494) - yikes! z* = 0.47588 (-0.0032)

May or may not be related to using the computer-controlled detector tilt feature? In my case I've always used it at a specific adjusted tilt value, while I think the service calibration was done in its "neutral" position ... will have to test this further. It's supposed to adjust the PC to take the altered detector tilt into account, but perhaps it's not doing a very good job at this.

[luqi-cloud]

Thanks for your kind attention. I also think the pattern centers provided by Aztec are not accurate. Now I use the pattern centers optimized by Genetic algorithm that has been implemented in AstroEBSD. After that the corresponding euler angles were calibrated by DI methiod. The resulting euler angles are (185.551,4.913,53.312) . The misorientation anndle is 1.608 (in degree) from original euler angles. PC from AstroEBSD (Genetic algorithm): x = 0.519 y = 0.67 z = 0.5111768

In this case, I don't think neural position has significant influences because the step size is quite small (150nm). step_x = 233, step_y = 250.

In addition, I think the pixel size is 22.91*2 because binning size is 2.

[Philip-Go]

I found a solution to the y-PC Calculation for Oxford EBSD Diffration Patterns. Maybe it is still of interest.

I have been struggling with Pattern Centers from Oxford/AZtec troughout the last 3-4 months. I was stubbling across this Issue, however did not manage to fit a Pattern Center e.g. with AstroEBSD.

However, when exporting experimental Kikuchi Patterns with AZtec it is possible to export them with the Pattern Center indicated with a cross. Measuring the location of that PC cross then gives the proper pixel position, which then can translated into the reference frame of EMsoft (origing in the middle of the detector screen). After several patterns and Pattern Centers assessed this it became obvious, that Oxford defines its pattern center in relation to the image WIDTH: y* = Vertical_distance(PC - HorzDown) / width(image) this way, the calculation of the PC for EMsoft follows:

yPC = Nx y - Ny*/2

This works currently on two systems equipped with Oxford Nordlys and Symmetry detectors for different acc. voltages with average misorientations of ~1° to the Hogh indexed map.

Thanks for your kind attention. I also think the pattern centers provided by Aztec are not accurate. Now I use the pattern centers optimized by Genetic algorithm that has been implemented in AstroEBSD. After that the corresponding euler angles were calibrated by DI methiod. The resulting euler angles are (185.551,4.913,53.312) . The misorientation anndle is 1.608 (in degree) from original euler angles. PC from AstroEBSD (Genetic algorithm): x = 0.519 y = 0.67 z = 0.5111768

In this case, I don't think neural position has significant influences because the step size is quite small (150nm). step_x = 233, step_y = 250.

In addition, I think the pixel size is 22.91*2 because binning size is 2.

To confirm this with your case, the initial value form your opening post is for y* = 0.548. The experimental pattern has a size of 622 x 512 px. From the formula above you get the vertical position in px:

Vertical_distance(PC - HorzDown) = y width(image) = 0.548 * 622 = 340.9

Normalize this by the height of the image and you get the value you determined via AstraEBSD:

Vertical_distance(PC - HorzDown) / height(image) = 340.9 / 512 = 0.666

So basically, the transformation convention for the Oxford system (at least in this thread and in my case) is pretty messed up.

[luqi-cloud]

Yes, it is right the Oxford defines its pattern center in relation to the image width. So I think the right way to calculate the pattern center is: y = y/Ny Nx = 0.5436/1024 1244 = 0.66. It is similar to the PC optimized by AstroEBSD. yPC = (y-0.5) Ny = 0.16*1024

Second, please note the pc convention adopted by AstroEBSD is different. The origin defined by AstroEBSD is at the top left corner,and the pcy is at the ratio of image height rather than image width. So the pcy optimized by AstroEBSD is around 0.33. Another trick for AstroEBSD is increasing the PopulationSize and MaxGenerations (in the script named EBSP_PCSearch) .

I found a solution to the y-PC Calculation for Oxford EBSD Diffration Patterns. Maybe it is still of interest.

I have been struggling with Pattern Centers from Oxford/AZtec troughout the last 3-4 months. I was stubbling across this Issue, however did not manage to fit a Pattern Center e.g. with AstroEBSD.

However, when exporting experimental Kikuchi Patterns with AZtec it is possible to export them with the Pattern Center indicated with a cross. Measuring the location of that PC cross then gives the proper pixel position, which then can translated into the reference frame of EMsoft (origing in the middle of the detector screen). After several patterns and Pattern Centers assessed this it became obvious, that Oxford defines its pattern center in relation to the image WIDTH: y* = Vertical_distance(PC - HorzDown) / width(image) this way, the calculation of the PC for EMsoft follows:

yPC = Nx y - Ny*/2

This works currently on two systems equipped with Oxford Nordlys and Symmetry detectors for different acc. voltages with average misorientations of ~1° to the Hogh indexed map.

Thanks for your kind attention. I also think the pattern centers provided by Aztec are not accurate. Now I use the pattern centers optimized by Genetic algorithm that has been implemented in AstroEBSD. After that the corresponding euler angles were calibrated by DI methiod. The resulting euler angles are (185.551,4.913,53.312) . The misorientation anndle is 1.608 (in degree) from original euler angles. PC from AstroEBSD (Genetic algorithm): x = 0.519 y = 0.67 z = 0.5111768 In this case, I don't think neural position has significant influences because the step size is quite small (150nm). step_x = 233, step_y = 250. In addition, I think the pixel size is 22.91*2 because binning size is 2.

To confirm this with your case, the initial value form your opening post is for y* = 0.548. The experimental pattern has a size of 622 x 512 px. From the formula above you get the vertical position in px:

Vertical_distance(PC - HorzDown) = y width(image) = 0.548 * 622 = 340.9

Normalize this by the height of the image and you get the value you determined via AstraEBSD:

Vertical_distance(PC - HorzDown) / height(image) = 340.9 / 512 = 0.666

So basically, the transformation convention for the Oxford system (at least in this thread and in my case) is pretty messed up.

[hakonanes]

Would just like to add here that @IMBalENce and I have implemented conversions between PC conventions used in Bruker, EDAX TSL, EMsoft (v4 and v5), and Oxford in the Python package kikuchipy (relevant documentation):

>>> import matplotlib.pyplot as plt
>>> import kikuchipy as kp
>>> p = plt.imread("ex.jpg")
>>> detector = kp.detectors.EBSDDetector(
...     shape=p.shape,
...     pc=(0.48, 0.54, 0.51),
...     px_size=50,
...     convention="oxford",
...     sample_tilt=70,
...     tilt=-2,
... )
>>> detector  # PC in Bruker convention used internally
EBSDDetector (512, 622), px_size 50 um, binning 1, tilt -2, azimuthal 0, pc (0.48, 0.344, 0.62)
>>> detector.pc_emsoft()
array([[1.2440e+01, 7.9880e+01, 1.5861e+04]], dtype=float32)  # (12.44, 79.88, 15861)
>>> detector.pc_oxford()  # pc_tsl() is identical
array([[0.48, 0.54, 0.51]])
>>> detector.plot(pattern=p)  # PC plotted as star

We believe we have implemented all conversions correctly: if you find we haven't, please open an issue here.

[gsparks3]

So it turns out that this whole issue is because Oxford apparently changed how the pattern center values are represented internally, but didn't tell anyone. I (and presumably everyone else who was having issues) was using the "Oxford" convention as defined in the DI tutorial. While I don't know exactly when the change occured, in more recent systems, it seems that the correct PC pixel values are obtained using the "EDAX" convention. This explains why only the Y-value seemed incorrect - the conversion equations for the other two components remain the same. The numerical values in the TIF tags are correct, they just need to be converted using the appropriate equations.

[luqi-cloud]

Yes, the PC given by Oxford (Aztec) are correct. They just need to be converted using the appropriate equations for DI. The PC convention adopted by Aztec has been clearly stated in the latest Aztec user guide. The origin is at the bottom left corner. All PC is at the ratio of image width.

So it turns out that this whole issue is because Oxford apparently changed how the pattern center values are represented internally, but didn't tell anyone. I (and presumably everyone else who was having issues) was using the "Oxford" convention as defined in the DI tutorial. While I don't know exactly when the change occured, in more recent systems, it seems that the correct PC pixel values are obtained using the "EDAX" convention. This explains why only the Y-value seemed incorrect - the conversion equations for the other two components remain the same. The numerical values in the TIF tags are correct, they just need to be converted using the appropriate equations.

[hakonanes]

Found an old Channel 5 manual from 2007 on the internet, which states the same PC x and y convention as in the AZtecCrystal User Guide (haven't used the Channel 5 software suite, but I understand that Flamenco is one of many softwares in the suite):