py4DSTEM not reading Prismatic output

[ahinroy]

Hi @bsavitzky, I want to load the output file from Prismatic simulation to py4DSTEM and apply the detector functions to calculate images for different collection angles. The output is not being read by py4DSTEM. Is it necessary for the HDF5 to have EMD attributes to be able to be read by py4DSTEM?

[lerandc]

an addendum to this: I tried this as well, and it looks like the error is related to "4DSTEM_simulation" not being passed in as the top group name

relevant stack trace:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
 in 
----> 1 py4DSTEM.file.io.read(fp)

~/.local/lib/python3.7/site-packages/py4DSTEM/file/io/read.py in read(fp, mem, binfactor, ft, **kwargs)
     70 
     71     if ft == "py4DSTEM":
---> 72         data,md = read_py4DSTEM(fp, mem=mem, binfactor=binfactor, **kwargs)
     73     elif ft == "dm":
     74         data,md = read_dm(fp, mem, binfactor, **kwargs)

~/.local/lib/python3.7/site-packages/py4DSTEM/file/io/native/read_py4DSTEM.py in read_py4DSTEM(fp, **kwargs)
     54     """
     55     assert(is_py4DSTEM_file(fp)), "Error: {} isn't recognized as a py4DSTEM file.".format(fp)
---> 56     version = get_py4DSTEM_version(fp)
     57     if not version_is_geq(version,(0,9,0)):
     58         if version == (0,6,0):

~/.local/lib/python3.7/site-packages/py4DSTEM/file/io/native/read_utils.py in get_py4DSTEM_version(fp, topgroup)
     32     assert(is_py4DSTEM_file(fp)), "Error: not recognized as a py4DSTEM file"
     33     with h5py.File(fp,'r') as f:
---> 34         version_major = int(f[topgroup].attrs['version_major'])
     35         version_minor = int(f[topgroup].attrs['version_minor'])
     36         if 'version_release' in f[topgroup].attrs.keys():

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

~/.local/lib/python3.7/site-packages/h5py/_hl/group.py in __getitem__(self, name)
    262                 raise ValueError("Invalid HDF5 object reference")
    263         else:
--> 264             oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
    265 
    266         otype = h5i.get_type(oid)

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5o.pyx in h5py.h5o.open()

KeyError: "Unable to open object (object '4DSTEM_experiment' doesn't exist)"

edit: perhaps in below, find top groups first, then check version? I don't think this would break things unless a single file has multiple versions in it at once

https://github.com/py4dstem/py4DSTEM/blob/00291ed0368cd491a22b4331e5643778fb4ffec6/py4DSTEM/file/io/native/read_py4DSTEM.py#L14-L78

[bsavitzky]

Fixed - updated and pushed to PyPi as version 0.9.11. @ahinroy, if you update with pip install py4dstem --upgrade, this should work now. Thanks @lerandc - your suggested fix does indeed do the trick!

[lerandc]

@bsavitzky forgot to comment here again (this discussion also came up on the prismatic gitter), just tested some things on my own some more, and it looks like it will still throw-- with the following error

Exception: Support for legacy v0.5.0 files has not been added yet. I have not yet tested if the legacy reader for v0.6 will work for the Prismatic sim outputs, so that might be an easy fix as well, but something in my distant memory is telling me that it might not...

[bsavitzky]

Ah, gotcha, thanks. The topgroup bug is now fixed, but I tested only this issue specifically. At some point we should consider having prismatic spit out a more up-to-date py4DSTEM file version, but we do also need to support the older formats, so order of operations here seems maybe not so important. Right now we're not even trying to read v0.5.0 files, just finding them and throwing that exception you mentioned - my guess is that all the datatypes other than PointLists and PointListArrays will read without trouble with the existing readers, but those last two might need a little finessing. Do you want to send me a small sample file that prismatic is spitting out to test on?

[lerandc]

with the next release I intend to align Prismatic to v0.9 specs (unless v0.10 or v1.x comes out before I get around to it?)-- that's a this week question for my dev schedule. will send some simdata shortly

[bsavitzky]

Hey @lerandc - going through old issues at last. Is this resolved? If not, do you have a dataset handy I can use to test against before we release v14.0?

[lerandc]

I believe this has been solved, though the current prismatic format, I believe, would be read in through one of the legacy data readers. I'll pass you some simulation data shortly

[bsavitzky]

Awesome, got the data, thank's Luis! I also suspect this will work in v14, but want to add in the proper tests to be sure - once those are added with the test files you sent and everything checks out, I'll close the issue