search

hyperspy / rosettasciio

Python library for reading and writing scientific data format
https://hyperspy.org/rosettasciio
GNU General Public License v3.0
47 stars 28 forks source link

Latest Velox 3.12 .emd files are not readable due to major changes in file format #226

Closed Philipp-Muller closed 6 months ago

Philipp-Muller commented 7 months ago

Velox 3.12 introduced changes in the .emd file format. The new files are not readable anymore.

https://www.fei-software-center.com/tem-apps/velox-download-page/

By now I can confirm that both STEM images and EDS elemental mappings are affected. I therefore assume it is a general issue.

I can provide example files within the next few days if required.

Thanks for the continuous support and development!

ericpre commented 7 months ago

Yes, that would be great if you could provide some files. Ideally, the smallest possible file so that they can be added to the test suite - if you are happy with that of course! 😃

Philipp-Muller commented 7 months ago

Disclaimer: The following files were acquired with velox 3.10 and then opend with Velox 3.12 (offline) this triggers the conversion to the new format.

Philipp-Muller commented 7 months ago

Au SI+Spectrum+Linescan 155 kx.zip several testfiles.zip Au SI 155 kx.zip

ericpre commented 7 months ago

It seems that the group "Operations" is missing from the file that you shared and it is used to parse information about detector/elemental map. This can be skipped but then we don't know what elemental map it is... or the type of DPC images, etc.

@robindegen, by any chance, are you still working on Thermo Fisher Velox EMD? If so, could you please advise on what happen to the "Operations" group? 😃

Philipp-Muller commented 7 months ago

If you require specific support from Thermo-Fisher please post here. The Velox developers follow the discussion.

shoving-tfs commented 7 months ago

If you require specific support from Thermo-Fisher please post here. The Velox developers follow the discussion.

Indeed we are following the discussion, Velox 3.12 introduced a file format change to support upcoming features.

Just for understanding, we call the 'pre 3.12 file format' the old file format. And the file format from Velox 3.12 and higher the new file format.

Important to understand The new file format only applies to Spectrum Imaging (SIFeature).

Here is a quick (incompleet) summary of the changes.

What changed

Feel free to ask any question.

ericpre commented 7 months ago

Thank you @shoving-tfs for the information! Can you advice on how to get the information about the images, for example, the element for EDS maps, or operation between detector quadrant for DPC data? I had a quick look at the new structure but couldn't find it what I was after and it would much more simple if you could provide some poiners.

shoving-tfs commented 7 months ago

Hi @ericpre thank you for your question, as I have started before only SIFeature has changed, DPC has not been effected.

Regarding the element for EDS Maps:

  1. /Displays/ImageDisplay contains the list of all the image displays. A EDS Map is just an image display.
  2. These entries contain a json encoded dictionary that contains 'data', 'id', 'settings' and 'title'.
  3. The 'id' is the name of the element. 'data' is pointing to the data reference in SharedProperties/ImageSeriesDataReference/<UUID> which in turn is pointing to the /Data/Image/<UUID> where the image data is located.

I hope this is enough to get you started.

ericpre commented 7 months ago

Thank you @shoving-tfs, that's exactly what was necessary! 😃 I was asking about DPC, because currently the Operations group is used to get information to know the type of DPC. However, from reading the file provided above, the code to read the label of EDS maps seems to be working well for DPC images in version 11.

@shoving-tfs, @Philipp-Muller, would you please be able to provide small test file to add the test suite in this repository? The files above are standard in size but this is too large to add to the repository, because we want to avoid the repository getting unnecessarily large.

This should be fixed in https://github.com/hyperspy/rosettasciio/pull/232 and we just need to add small test files.

ericpre commented 7 months ago

@shoving-tfs, @Philipp-Muller, I added some detailed explanation on the test data files in #233. See documentation output at https://rosettasciio--233.org.readthedocs.build/en/233/contributing.html#making-test-data-files. Any feedback (something unclear, ambiguous, etc) on this section is more than welcome!

ThorstenBASF commented 7 months ago

Attached you will find some EMD files for testing. Smallest size for imaging is 256x256. As last time, the files are acquired with Velox 3.10 and converted with 3.12. You also find a double converted BCF-File (BCD>EMD>EMD). Till now this file type is not supported, the error said something about a "prune" format. Velox_3.12_testfiles.zip

Thanks for your support!

ericpre commented 7 months ago

Thank you @ThorstenBASF for sharing the files. Is the "Test SI 16x16 ReducedData 215 kx.emd" file also one of these pruned file?

With the "Test BCF SI 64x64 converted", there is indeed a warning saying the spectrum image can't be read because "pruned files" are not supported and the warning suggest the user not to prune the file! ;) This is expected.

Philipp-Muller commented 7 months ago

The pruning is not a user decision. This is the file format which is generated by velox with the Bruker .bcf to .emd converter. "Test BCF SI 64x64 converted" is the only file of this type.

"Test SI 16x16 ReducedData 215 kx.emd" means that the file size reduction of velox was used. after that you cannot trim the individual frames of the eds acquisition. This is a user decision and could be avoided.

ericpre commented 7 months ago

Okay, then it sounds like this is saving the data in the same way. An alternative to the conversion is to load the bcf file directly using hyperspy. In any case, it would be good to improve the warning.

For reference, here is the original discussion for the prune files: https://github.com/hyperspy/hyperspy/pull/1662#issuecomment-361837198.