loadtxt, savetxt

[jlaehne]

I want to implement a simple set of wrappers that use np.loadtxt and np.savetxt to load and save txt, dat, csv format files. Mostly metadata agnostic of course. At least for single spectra as well as linescans. For more than one navigation dimension it would become more complicated, but at least 2d navigation with 0d signal axis (e.g. single parameters from fit results) would also be feasible. It would be helpful to import/transfer data from systems where the native file system is so far not implemented in hyperspy (e.g. Jobim Yvon spectrometer software or systems with self-programmed data collection), but a text format export is possible. On the writer side, I often want to pass on some data to colleagues who do not use hyperspy or even python (the good old Origin crowd).

@ericpre, @dnjohnstone, @jordiferrero would it make more sense to implement that in the io-part of hyperspy for general usage? Otherwise I would implement a specific set of functions just for lumispy?

[ericpre]

I want to implement a simple set of wrappers that use np.loadtxt and np.savetxt to load and save txt, dat, csv format files. Mostly metadata agnostic of course.

The issue with having a generic io plugin for txt/csv files is that there is no specification and therefore the plugin can't do much:

• loading data is trivial
• nothing to be done with restructuring the data
• not possible to set axes
• not possible to add metadata.

It would be helpful to import/transfer data from systems where the native file system is so far not implemented in hyperspy (e.g. Jobim Yvon spectrometer software or systems with self-programmed data collection), but a text format export is possible.

If there are specifications or it follows a specific know pattern, then this is different from the case above and, it makes sense to have a plugin - see for example msa.

[ericpre]

Having examples would be good, see https://github.com/hyperspy/hyperspy/pull/2306/files#diff-82b6f06aa6e3647b596d6dc24789557125d85d87ce203ad9a660ffc5c5ec78f4

[jlaehne]

I see your point with the 'specs' and it is true that for single spectra msa does the job. At least for saving I have my wrapper routines in a local script and think a generalized version could be nice to have in lumispy for others to use.

For the examples, I'll maybe add one for reading linescan data in txt format.

[jordiferrero]

I agree that we need some sort of function to integrate better with the Origin crowd. Gunnar (here in Cambridge) has asked me for this functionality many times, as collaborators may not use Python.

What about having some general functions in the luminescence class to export the data? I envision some very simple function that simply takes the s.data and the s.axes_manager.signal_axes[0].axis and puts it in a simple .csv file that Origin can load easily? Would that be fine? Or are you thinking something more complex?

And as for the "examples", would it make sense to have it in the lumispy-demos section? Or within the lumispy package?

[jlaehne]

I can contribute such a function, as I basically have some functions doing the job. I would just generalize them to decide the format based on the dimensions within the function instead of having several different functions:

• Spectrum: simple .csv (with options to choose the delimeter, maybe also to add a simple header)
• Linescan: signal axis as first column and navigation axis as first row
• 2D navigation, 0D signal (e.g. single attributes from a fit): simple matrix (with option to put axes in first row and column)
• 2D: Could in principle be represented by consecutive matrices. Do we need that? It is usually no fun to handle multidimensional data in Origin, so up to now I did not pass that type of data to anyone in a text format. I would just raise an error if dimensions exceed the three cases above.

I'll do it for LumiSpectrum for the moment. For LumiTransient, I would create a separate function eventually (alternative would be to generalize the above function and then put it in CommonLumi, but I think a separate function would be clearer).

• Basically, a single point dataset could be represented the same way as a spectral linescan above
• It could be interesting to save fit results, where one or multiple lifetimes are added to a simple .csv file
• Anything of higher dimension

[jlaehne]

I think the "examples" was more as an alternative for the HyperSpy case, where one could add an example for linescans. I think that is independent of whether we implement it here.

Apart from that, I would eventually include this functionality in one of the demos on lumispy-demos.

[jordiferrero]

I'm happy for you to go ahead. I also have some code for the 2D fitting results to pandas arrays. But please go ahead.

I absolutely agree that Origin is a big no for more than linescans. So just raise an error indeed. Let me know if you want me to help out with this eventually.

[gkusch]

As a comment, before you go too deep into certain directions. Origin can without problems load hdf5 files and is how I am handling data export at the moment in my local scripts for extracting spot spectra. hdf5 in this case would have the advantage that you can just append more and more points to it (should make a linescan easy) and can at least theoretically store metadata (I haven't had time to go that much into detail on the hdf5 file format...which is mre or less my story with all of this, not having enough time). In case I am not completely misreading what you would like, I am happy to send you the bit I wrote @jordiferrero for you to look at and potentially implement here.

[jlaehne]

I often have colleagues who want the data in a simple text format, that is why I have written some small functions. Nothing elaborate, but the main point is in putting the different vectors and the data matrix together the correct way.

I can put together a rough proposal and let you comment along the way before adding all the nice-to-have details such as choosing the delimiter.

If Origin can handle hdf5, all the better, as it is the native format of hyperspy so no specific functions needed for export, I guess - but I fear there are some colleagues who will still prefer a simple text format.

[jordiferrero]

I agree with both of you. Then I wait for your proposal and will comment on that.

[jlaehne]

See #56 for a the idea of implementation.