search

pyxem / kikuchipy

Toolbox for analysis of electron backscatter diffraction (EBSD) patterns
https://kikuchipy.org
GNU General Public License v3.0
82 stars 29 forks source link

Add option to load either processed or unprocessed from Oxford Instrument's H5OINA file #692

Closed hakonanes closed 2 hours ago

hakonanes commented 1 month ago

So about importing .h5oina data in Kikuchipy, it seems that the Dataname 'Processed Patterns' is still valid. But once there are also Unprocessed Patterns stored in the h5, it seems to me that the Kikuchipy reader starts to load these into memory, even when Lazy=True. Does the reader by default import all other data into memory, accept for the 'Processed Patterns', when Lazy=True?

Sorry for the spam, but I solved this issue, at least temporarily. I found the line that specifies that the pattern dataset should not be read into memory (I think), and added "Unprocessed Patterns" to it. I changed line 99 in oxford_h5ebsd.py to the following: dd = _hdf5group2dict(group["EBSD/Data"], data_dset_names=[self.patterns_name, "Unprocessed Patterns"]) This solves the issue for me. Perhaps this is not a suitable permanent solution though. I guess the user would need to have the option to import unprocessed patterns, if they like, instead of processed patterns...

Originally posted by @Tijmenvermeij in https://github.com/pyxem/kikuchipy/issues/690#issuecomment-2405719172

I did not think about H5OINA files storing both processed and unprocessed patterns when writing the reader. Thank you for spotting the bug, @Tijmenvermeij!

The solution is, as you say, to give the user the opportunity to load either of them (or both, if they load consecutively). The other, if present, should not be read into memory.

We should look at releasing a ~v0.11.0~ 0.12.0 within the next couple weeks with these changes.

Tijmenvermeij commented 1 month ago

For loading "Unprocessed Patterns" from .h5oina files, it probably also makes sense that the static BG is also loaded from the file. I'm not sure if this is currently done already...

hakonanes commented 1 month ago

The static background is loaded

https://github.com/pyxem/kikuchipy/blob/a016c0a38b84cc74b4110c517807504e5f248c50/src/kikuchipy/io/plugins/oxford_h5ebsd.py#L140-L141

and available in the property EBSD.static_background. This is also mentioned in the IO user guide.

hakonanes commented 1 week ago

@Tijmenvermeij, I made a fix for the processed/unprocessed patterns from H5OINA in https://github.com/hakonanes/kikuchipy/tree/692-h5oina-un-processed-patterns and #702.

Could you try it out? python -m pip install kikuchipy@git+https://github.com/hakonanes/kikuchipy.git@692-h5oina-un-processed-patterns

hakonanes commented 2 hours ago

Should be fixed in #702.