Not possible to save array of data (like a scope trace for example) on every iteration

[benmetcalf42]

I have a procedure object which on every iteration captures a full oscilloscope trace containing 2452 separate data points. Additionally, I capture this trace 11 separate times on each iteration for averaging purposes - this gives me a 2D (11*2452) numpy array that I wish to save under a single column ('Scope Trace') on each iteration of my procedure

However - the results object does not seem capable of handling anything other than a single value for each data_column specified

When I pass my 2D numpy array (112452) to the data variable in my procedure definition - I see this gets transformed to a literal string in the results.format method which literally writes out what is displayed in the python console when a large array is printed - i.e. just the first and last 3 values from the array separated by ellipses - this in turn is written to the .csv file which means only 6 out of the 112452 values are saved each time

Is there any way to modify the results class to be able to handle arbitrary sized numpy arrays contained within each Data Column? Even better would be to have the option of saving data to a pandas dataframe

I have only just started exploring PyMeasure though so I may have missed something obvious

[cjermain]

@benmetcalf42, the Results object is currently limited to a single value per data column. This limitation comes from the way the Results are written to a file, since this is done to the CSV format. If we introduce additional methods/formats for saving the data, then this limitation is not necessary.

I think this would be a great feature. Do you want to spec out what this feature would look like?

Check out Results._data and the Results.data property. This is a pandas DataFrame where the data resides.

[bilderbuchi]

Maybe it would make sense to more fully embrace pandas when processing results/data? The dataframe is well-suited for the time-based aquisition of columns of data, and you could leverage pandas' considerable I/O capabilties without reinventing the wheel.

[cjermain]

@bilderbuchi, I agree. However, we need to retain the header information that provides the Procedure information. So perhaps we want to look at HDF5, which has key value storage that could serve this purpose. Does anyone want to champion this feature of expanding the I/O capabilities of the Results object?

[bilderbuchi]

I hope that some day pandas will finally get metadata storage in a data frame. There's a much-discussed issue on their tracker, but it won't go far without outside contributions, apparently.

[jmittelstaedt]

A possible solution to this would be to use the data structures in the xarray package. The Dataset defined in that package is essentially a multi-dimensional version of a pandas DataFrame which supports arbitrary metadata in attrs, which is just an OrderedDict. The external file type preferred by xarray is the netCDF file format which supports such multidimensional data, but looking into other ways of externally storing data may be preferred since netCDF is not human readable. I think xarray Datasets would be a good internal representation for the data though, which would also work well for inherently multidimensional data e.g. saving oscillator traces or images as some other parameter is varied.

[jrr1984]

Has this been solved? Im trying to save a trace of an spectrometer (a list of nearly 2500 points) but it wont work with the plotter

[cjermain]

@jrr1984, this is still an open issue.

[PkRPintu]

Hi all,

Is the issue resolved already? I also would like to save the trace of the spectrum analyzer which is also an array. Could someone please let me know how this can be possible? I see in the instrument class there are libraries for many different spectrum analyzer, so will simply guess that many people are somehow accessing and saving the data from the spectrum analyzers by using pymeasure. I am new to pymeasure and do not have much ideas about all the utilities and know-how of it.

Many thanks for the help.

Best wishes Pradeep

[BenediktBurger]

Hi @PkRPintu ,

sorry for the late reply.

Pymeasure consists of two parts, a measurement setup with a GUI and an instrument driver part. The limitation talked about in this issue resides in the measurement setup. This setup cannot take a series of traces at the moment. The spectrum analyzer istruments, however, are capable of transferring full spectra. That means, that you have to write your own measurement software (for example a loop), to read the spectrum analyzer. You might want to have a look into PyMoDAQ, which offers the possibility to take a series of spectra. You can, nevertheless, use pymeasure's instrument driver.

I hope this helps. If you have questions, don't hesitate to ask.