Loading EDAX EBSD data

[dtfullwood]

I gather I can load either cpr+crc file or a ctf file from Oxford / Aztec. It looks like there is also an option for a python-based database. What are people regularly doing to load EDAX type data? (.ang files) I can probably convert from ang to ctf format fairly easily. Would it be easier to go to the python database option? If so, do you have an example of a format? Thanks David

[mikesmic]

Hi David, the best option is to write a reader for the EDAX file format in defdap. It shouldn't be too difficult, I knew this day would come so I made a seam between loading in data and creating the map objects. Is there any documentation for the ang file format and do you have an example small map? The tricky bit will be sorting the reference frames. Currently we use the convention set for the oxford systems and this is fixed. We can either make the convention variable in defdap or convert EDAX data as it is loaded (probably easier). Another issue will be if you use the hex grid layout, I'm not sure what would be best to do about that.

[dtfullwood]

Thanks - I have sent a .ang file and some other info to your Manchester email address. I will copy the email and files here:

Here is a typical .ang file (of some typical steel), and the matlab file that OpenXY uses to read it (which may be useful for reading the header info). The format of the data in the main data section is: 1-3: Euler angles 4: x position (in default EDAX sample frame – not in Euler angle frame) 5: y position 6: Image Quality 7: Confidence Index 8: Phase number 9: ? 10: Fit factor

To change from Oxford Euler angle frame to EDAX Euler angle frame we simply add pi/2 to the first Euler angle. Hence, to go from EDAX to Oxford you would need to subtract pi/2 from the first Euler angle (see the attached powerpoint). The default EDAX Euler angle reference frames is the same for any user sample reference frame selected, I believe. The default sample frame in EDAX is #2. See the attached ‘reference frames.pptx’ It may be best to just insist that people use square grid scans in the short term? Also, the Pattern Center system is different between Oxford and EDAX. In EDAX, xstar, ystar and zstar are in fractions of the phosphor screen. In Oxford the parameters are PCX, PCY, VHRatio (vertical to horizontal ratio of EBSP) and DD (detector distance) Then: xstar=(PCX-(1-VHRatio)/2)/VHRatio ystar=PCY/VHRatio zstar=DD/VHRatio I’m copying Matt and Stuart at EDAX to fill in the missing #9 column info, and in case they have any other comments on converting from EDAX to Oxford for your HRDIC cod Archive 3.zip e.

[hakonanes]

the best option is to write a reader for the EDAX file format in defdap.

We have readers for .ang files (from EDAX TSL, NanoMegas ASTAR Index and EMsoft's EMgetANG program) in orix (documentation, code). To not duplicate efforts, it would be good if we could use orix and DefDap both in the same environment, say use orix to read data and DefDap for further analysis?

>>> from orix import io
>>> xmap = io.load("OIM Map 2.ang")
/home/hakon/kode/orix/orix/io/plugins/ang.py:267: UserWarning: Number of columns, 10, in the file is not equal to the expected number of columns, 14, for the 
assumed vendor 'tsl'. Will therefore assume the following columns: euler1, euler2, euler3, x, y, unknown1, unknown2, phase_id, unknown3, unknown4, etc.
  warnings.warn(
>>> xmap
Phase    Orientations  Name  Space group  Point group  Proper point group     Color
    0  17017 (100.0%)  None         None         None                None  tab:blue
Properties: unknown1, unknown2, unknown3, unknown4
Scan unit: nm
>>> xmap.rotations.to_euler()  # Euler angles array of shape (n pixels, 3)
array([[-0.42987531,  0.46803   ,  1.45128   ],
       [-2.10708531,  1.51824   , -0.46453531],
       [-0.42645531,  0.47031   ,  1.44969   ],
       ...,
       [ 1.34187   ,  2.36922   , -0.18069531],
       [ 1.34831   ,  2.36557   , -1.73153531],
       [ 1.37941   ,  2.35087   , -1.71272531]])
>>> xmap.unknown1
array([7444.7, 7398.8, 7428.1, ..., 6823.3, 6806.2, 7062.3])

The IO reader returns an orix CrystalMap instance (similar to MTEX' EBSD class), but of course all data is available as NumPy arrays which can be inserted into an object DefDap can work with.

An alternative way to rotate the orientations from EDAX' (setting 2, RD-TD-ND) to Oxford's/Bruker's sample reference frame is (continuing the example above)

>>> import numpy as np
>>> from orix.vector import AxAngle, Vector3d
>>> from orix.quaternion import Rotation
>>> axangle_tsl2oxford = AxAngle.from_axes_angles(Vector3d.zvector(), -np.pi / 2)
>>> r_tsl2oxford = Rotation.from_neo_euler(axangle_tsl2oxford)
>>> rot = r_tsl2oxford * xmap.rotations
>>> rot.to_euler()
array([[-0.42987531,  0.46803   ,  3.02207633],
       [-2.10708531,  1.51824   ,  1.10626102],
       [-0.42645531,  0.47031   ,  3.02048633],
       ...,
       [ 1.34187   ,  2.36922   ,  1.39010102],
       [ 1.34831   ,  2.36557   , -0.16073898],
       [ 1.37941   ,  2.35087   , -0.14192898]])

orix doesn't support hexagonal grids either, we have an open issue discussing this feature.

[dtfullwood]

Thanks Hakon, Michael wrote a reader for .ang files in the 'develop' branch over the weekend. It uses this syntax: ebsd_file_path = '/Users/mbcx9ma4/Downloads/OIM Map 2' ebsd_map = ebsd.Map(ebsd_file_path, dataType='EdaxAng') We tested it out yesterday and it works for regular .ang files. If you crop a .ang file using OIM, it removes the blank comment lines between phase information segments, which the reader depends on to detect the next segment. So you need to add them back in (i.e. add a line with just a '#' after each segment for each phase). It also flags an error if you try hexagonal grid scans since it only works for square scans (which is all we generally used for HREBSD anyway). I guess we will need plotting functions for the image quality to help with homologous point identification next, which should be fairly straightforward. So big thanks to Michael. Thanks again for your tips, David

[hakonanes]

Michael wrote a reader for .ang files in the 'develop' branch over the weekend.

Good to hear that you can read your data!

Seeing that an EDAX reader is already implemented in DefDap, we should at least take inspiration from each other's readers to make them both more robust. For example, I see that the phase information couldn't be read from your .ang file into orix... I'll try to improve that myself by debugging with your file, thanks.

I guess we will need plotting functions for the image quality to help with homologous point identification next, which should be fairly straightforward.

Note that orix has a nice CrystalMap.plot() (pass any property name as a string, a NumPy array, etc. (documentation)) method that wraps Matplotlib's imshow(). Seems like DefDap has something similar.

[hakonanes]

If implementing readers for other formats comes up in the future, here's the list of readers in orix.

[mikesmic]

Hi Hakon

I've had a look through orix, it looks like a nice package you have put together. It would be good to look at using some of the structures you've put together within defdap, you definitely have a more thorough treatment of crystal structures and symmetry. But to do this properly would be a major rewrite, which I can't see me having time for in the near future.

David: I'll get back to you on swapping over the data used in the homologous point picker

[hakonanes]

It would be good to look at using some of the structures you've put together within defdap, you definitely have a more thorough treatment of crystal structures and symmetry. But to do this properly would be a major rewrite, which I can't see me having time for in the near future.

The same goes for orix, as DefDap has some nice plotting features and calculations that we would like to support as well.

My motivation for commenting on this issue is just to reduce the duplication of efforts in the realm of analysis of crystal orientations from mapping data in Python, and encourage us to learn from each other.

To that end, we keep a list of related projects in orix, which I'll add DefDap to soon (it's already listed in kikuchipy's list). If it's OK with you, I'll make a PR soon with a similar list to DefDap's documentation, listing both kikuchipy and orix :)

[mikesmic]

I agree completely, no point in doing things twice and better to spend time on the focus of the package. There seems to be a lot more python packages in the area of orientation mapping now, MTEX was the only real option when I got started and although it is a great package, I'm not a fan of MATLAB. It would be great to have more of a community around all these different packages and I welcome your pull request.

I do like the idea of using a more complete treatment of crystal symmetry in defdap, we have a somewhat noddy treatment in areas but it supports what we need it to. I would need to get some experience with the data structures you have created first. The best way to do that would be porting over the oxford file loaders I have over to orix, I will try and look at doing this in the next couple of months.

[hakonanes]

There seems to be a lot more python packages in the area of orientation mapping now, MTEX was the only real option when I got started and although it is a great package, I'm not a fan of MATLAB. It would be great to have more of a community around all these different packages and I welcome your pull request.

That is great to hear! I share your view of MTEX.

I do like the idea of using a more complete treatment of crystal symmetry in defdap, we have a somewhat noddy treatment in areas but it supports what we need it to. I would need to get some experience with the data structures you have created first.

orix has already supported nearly all of the crystal symmetry operations in pyxem, diffsims and kikuchipy for more than a year, so it is a somewhat tried and tested package. However, we would very much like for other packages to use it as well to make it more robust.

The best way to do that would be porting over the oxford file loaders I have over to orix, I will try and look at doing this in the next couple of months.

I agree, this makes the most sense. This is a feature I would very much like to use in kikuchipy as well, so that we could read both patterns from an .ebsp file together with the corresponding indexing data. I would be happy to help with this. Also note that the IO setup in orix isn't perfect, so any thoughts on improving the general setup is appreciated.

If defdap were to depend on orix for IO, it would make sense to create a converter from a CrystalMap to you EBSD class in defdap. I could help with this as well.

[rhysgt]

Closing as we now have a working EDAX loader. When we have capacity to look at better integration with orix we should make a separate issue or discussion.