Implement dictionaries for power and temperature sweep data

[crmcrae]

I propose we use Python dictionaries to organize sets of data for various powers and temperatures, which our TLS fitting and power/temp sweep plotting tools would then read in. What do you think @mullinska ?

[mullinska]

I'm not really sure what you mean by using python dictionaries to organize sets of data. Are you saying dynamically during fitting it will keep each data set in a dictionary? If so I don't believe that's necessary because all we really need are the Qi and Qc values from each iteration.

[joshmutus]

@crmcrae I'm a big fan of using data structures like this when we can. It will simplify the code and make it more readable. To add to that, there's a special thing called a dataclass that is like a dictionary but even more useable that we can use here.

For example we can create a DataSet class like:

@dataclass
class DataSet:
  freqs: Array
  amplitudes: Array
  phases: Array

then when we want to put the information in the dataset we do

new_data = DataSet(freqs = [....], ampliudes = [....], phases = [....])

and we can now access all the attributes really simply, for example when we plot amplitude vs frequency we can write:

plt.plot(new_data.freqs, new_data.amplitudes)

[FaustinCarter]

In the case where your data is multiple vectors that share a common coordinate vector (i.e. frequency is the coordinate vector, and phase, magnitude, I, Q, etc are the data vectors) then either a pandas dataframe or an xarray dataset seem like more reasonable options (both provide built in hooks to plotting, broadcasting for element-by-element operations, slicing, etc etc). Xarray is especially nice because you can have multiple coordinates (i.e. temperature, power, magnetic field, whatever) and you can slice across them more or less in any way you like. You can also then store those datasets in a compact binary representation (netcdf4 or hdf5 format). You also get dot-completion access for both (data.freqs, data.I, data.phase, etc).

-FC

On Wed, Mar 11, 2020 at 2:57 PM Josh Mutus notifications@github.com wrote:

@crmcrae https://github.com/crmcrae I'm a big fan of using data structures like this when we can. It will simplify the code and make it more readable. To add to that, there's a special thing called a dataclass https://docs.python.org/3/library/dataclasses.html that is like a dictionary but even more useable that we can use here.

For example we can create a DataSet class like:

@dataclassclass DataSet: freqs: Array amplitudes: Array phases: Array

then when we want to put the information in the dataset we do

new_data = DataSet(freqs = [....], ampliudes = [....], phases = [....])

and we can now access all the attributes really simply, for example when we plot amplitude vs frequency we can write:

plt.plot(new_data.freqs, new_data.amplitudes)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Boulder-Cryogenic-Quantum-Testbed/measurement/issues/58#issuecomment-597898019, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACTQRRH7MGFNO32G3HQNIBDRHACLXANCNFSM4KRX3R2Q .

[crmcrae]

That sounds great, thanks Faustin! @mullinska do you have a preference between pandas and xarray?

[mullinska]

I have used pandas before, but I'm sure that xarray isn't that hard to figure out so I really don't have a strong preference either way. As it stands right now the data is just being stored in numpy arrays inside of a data keeping class.

[FaustinCarter]

In my resonator analysis package (scraps) I generally used pandas for storing fit parameters as functions of temperature and input power. Once xarray became minimally stable I also started rewriting the entire thing to use xarray as the base data store with all the class methods as properties on top of that. I never really finished integrating it accross the whole package, but the resonator class code is here: https://github.com/FaustinCarter/scraps/blob/xarray/scraps/resonator.py and you can feel free to borrow as much as you like.

In general xarray is an N-dimensional generalisation of a pandas series (in much the same way that a DataFrame is a 2D series). I'm not sure what the right choice is, but xarray has some nice features for storing metadata (in pandas that's not really supported), concatenating data along any axis, dumping out to a usable binary form, etc.

I think pandas probably has a simpler interface, but you can always dump any two dimensions from an xarray out to a pandas dataframe with minimal effort (and vice versa).

The downside is that they are developing very actively and sometimes features get deprecated, or breaking changes are introduced.

I went with xarray because I wanted to support an arbitrary number of "knobs" like magnetic field, temperature, VNA power, etc. and I really wanted to store all my metadata (IF bandwidth, VNA model, resonator material, etc) in the same file as my data (in a compact, standardized binary format).

-FC

On Tue, Mar 17, 2020 at 1:41 PM mullinska notifications@github.com wrote:

I have used pandas before, but I'm sure that xarray isn't that hard to figure out so I really don't have a strong preference either way. As it stands right now the data is just being stored in numpy arrays inside of a data keeping class.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Boulder-Cryogenic-Quantum-Testbed/measurement/issues/58#issuecomment-600286739, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACTQRRHL6TYWUYCZZQHG6TDRH7OA3ANCNFSM4KRX3R2Q .

[joshmutus]

Hi @FaustinCarter xarray looks like a really interesting option.

[joshmutus]

@FaustinCarter would you be interested in porting over your data structure into this repo? It might be a nice contained project.

[FaustinCarter]

Hey @joshmutus, I'm ok with this idea, but it's not clear to me exactly how you want it implemented. Is the idea that this would replace the Resonator class currently located here by using the guts of scraps (xarray based storage with propery-based methods) and adding in the necessary hooks for cryores to use it:

https://github.com/Boulder-Cryogenic-Quantum-Testbed/measurement/blob/ddf922d6717e46c6fcb72454cc597149dfcb29a2/resfit/fit_resonator/resonator.py#L9-L20

Or did you want it implemented in some other way? Another option is that I would be willing to put some work into refactoring and extending the scraps code so that you could just import the scraps class directly (i.e. making scraps a dependency). This might make future updating easier as there would then be a single source for the code. Let me know what you think and I can try and figure out how to go about it.

It might take a bit of time since I'd mostly have to work on it in my spare time.

[mullinska]

Hi @FaustinCarter I saw this post so I thought I would clear up that in the last major push that @joshmutus made, the data is no longer stored in the Resonator class. The current place the data is being stored is:

https://github.com/Boulder-Cryogenic-Quantum-Testbed/measurement/blob/ddf922d6717e46c6fcb72454cc597149dfcb29a2/resfit/fit_resonator/fit_S_data.py#L783-L808

Where there is an option to read in the data from a .csv file or to feed the data directly into the function. The old place the data was being stored (the Resonator class) isn't really being used for anything at the moment, so if the data struct were to be implemented it would need to take over this function.