qt3scan: improve scan saves

[gadamc]

Confocal scans should be saved with more information than is currently saved.

Presently we only save the measured count rates in a simple 2D numpy array.

Ideally, we should save the following per stage position

• raw number of counts
• count rate

into a 2D array.

Additionally, at a minimum we should save the scan range and resolution. And we could envision the usefulness of also saving other data acquisition parameters, such as clock sample rate, and number of clock samples per position.

There is also a push to save data in hdfs format - see issue #67 - which we could consider doing in this ticket.

To accomplish these goals we would need to

• modify NiDaqDigitalInputRateCounter without side-effects on qt3scope and the experiment classes such that the raw counts can be returned as well as the count rate for each scan position
• modify the piezoScanner so that it holds both the raw data counts and the count rate (or perhaps, equivalently, the raw data counts and the sample time, allowing for easy computation of the count rate)
• ScanImage class in piezoscan.py holds the necessary data
• add h5py as requirement
• update piezoscan.py save data function to package data into h5py
• need some kind of documentation and/or class to define our h5py data structure(?)

If we work on this issue, then Issue #67 and Issue #35 should be closed. Or Issue #67 can be re-written just to define the work to incorporate the qt3experimentmanager tool.

[gadamc]

An alternative is not to use h5py but instead serialize to JSON and native python data types. @mfparsons314 what are the reasons to move to h5py?

[mfparsons314]

Some reasons, not sure that all are valid and happy to discuss more:

• I'm not sure serialization is a great option for large numerical datasets (imagine a 3d confocal scan with a spectrum at each point-- 4d data), without compression they may be quite large compared to binary hdf files
• HDF is supposed to be faster than serialization for large datasets
• I like the metadata option and tree structure of hdf, it allows one to nicely store parameters used to acquire data associated with results data and also to keep multiple datasets associated with the same sample in one file, for example, although JSON may be similar, not sure if serialization allows for a tree structure easily?
• my understanding is that HDF allows you to simultaneously read one part of a file and write to a different part, similar to a database
• hdf is portable across programming languages and operating systems-- I think probably JSON is as well so maybe that's ok, but pickle is not Also some discussion of options in this stackexchange https://datascience.stackexchange.com/questions/262/what-are-the-advantages-of-hdf-compared-to-alternative-formats

On Tue, Jan 31, 2023 at 5:42 PM G Adam Cox @.***> wrote:

An alternative is not to use h5py but instead serialize to JSON and native python data types. @mfparsons314 https://urldefense.com/v3/__https://github.com/mfparsons314__;!!K-Hz7m0Vt54!k1HS6_WEOaEY5w0cuGn8Twg0qfOQY-R5YW9G-g2XnV-PxZeH3JwaMutfo7GMGPi1MJdNC1cRwsTCyB4EMby8Kw$ what are the reasons to move to h5py?

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/qt3uw/qt3-utils/issues/76*issuecomment-1411323874__;Iw!!K-Hz7m0Vt54!k1HS6_WEOaEY5w0cuGn8Twg0qfOQY-R5YW9G-g2XnV-PxZeH3JwaMutfo7GMGPi1MJdNC1cRwsTCyB6i5iynWA$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A4B4KHBDUMPPLPV56HRM5KTWVG5QBANCNFSM6AAAAAAUNDX2E4__;!!K-Hz7m0Vt54!k1HS6_WEOaEY5w0cuGn8Twg0qfOQY-R5YW9G-g2XnV-PxZeH3JwaMutfo7GMGPi1MJdNC1cRwsTCyB7qJYxOog$ . You are receiving this because you were mentioned.Message ID: @.***>

--

Max Parsons

(he/him) Director, Quantum Technologies Training and Testbed (QT3) Laboratory https://sites.google.com/uw.edu/qt3-lab/home University of Washington, Dept. of Electrical and Computer Engineering

(206) 543-9689

[gadamc]

@mfparsons314 do you envision using hdf5 file format in a way that would store ALL of the data for a single sample? That is, in a single file you'll have multiple confocal scans and multiple spectra taken over the lifetime of that sample?

[mfparsons314]

Not necessarily, and this could be left up to the user. In the past, I've used a lot of different combinations-- 1 HDF per scan, 1 HDF per day, 1 HDF per sample. It all works pretty well.

On Wed, Feb 1, 2023 at 9:52 AM G Adam Cox @.***> wrote:

@mfparsons314 https://urldefense.com/v3/__https://github.com/mfparsons314__;!!K-Hz7m0Vt54!gs09Qc7CTZc5jqYud6Vs4rvl3O6I2A0seUhebwhXJJ22qg1Tz6tVxzkRBQUVG7NwS_anGmRnqjNlCKQ6_bvvTg$ do you envision using hdf5 file format in a way that would store ALL of the data for a single sample? That is, in a single file you'll have multiple confocal scans and multiple spectra taken over the lifetime of that sample?

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/qt3uw/qt3-utils/issues/76*issuecomment-1412480829__;Iw!!K-Hz7m0Vt54!gs09Qc7CTZc5jqYud6Vs4rvl3O6I2A0seUhebwhXJJ22qg1Tz6tVxzkRBQUVG7NwS_anGmRnqjNlCKTyav56XA$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A4B4KHB7A52BNELNEGTA5O3WVKPHPANCNFSM6AAAAAAUNDX2E4__;!!K-Hz7m0Vt54!gs09Qc7CTZc5jqYud6Vs4rvl3O6I2A0seUhebwhXJJ22qg1Tz6tVxzkRBQUVG7NwS_anGmRnqjNlCKQsrpN7rw$ . You are receiving this because you were mentioned.Message ID: @.***>

--

Max Parsons

(he/him) Director, Quantum Technologies Training and Testbed (QT3) Laboratory https://sites.google.com/uw.edu/qt3-lab/home University of Washington, Dept. of Electrical and Computer Engineering

(206) 543-9689

[gadamc]

This is currently in PR #81

[gadamc]

fixed in PR #81