search

QCoDeS / Qcodes_loop

Loop and matching dataset that used to be part of qcodes
Other
1 stars 1 forks source link

qcodes dataset cannot be stored easily #23

Open eendebakpt opened 7 years ago

eendebakpt commented 7 years ago

The qcodes dataset can only be stored to disk using a qcodes formatter. To save results totgether with other data it would be good to be able to serialize a dataset to a string or something like pickle, json or xml. So someting like:

s=dataset.tostring()
dataset2 = qcodes.load_dataset_string(s)

or

mystring=pickle.dumps(dataset)
dataset2=pickle.loads(mystring)

The method used for converting the dataset to a string could be taken from the formatter.

giulioungaretti commented 7 years ago

yes that is a problem. The entire data_set needs more love. I am thinking and thinkering with sqlite. This should allow us to have a better life 🥇

giulioungaretti commented 7 years ago

@eendebakpt @AdriaanRol @damazter @MerlinSmiles

now the big question is:

MerlinSmiles commented 7 years ago

@giulioungaretti i'd like it to be independent(no funny software to install to run a database just to get a single trace, and fast as in saving and loading a dataset. And i want to be able to load that data in 15 years from now.

giulioungaretti commented 7 years ago

@MerlinSmiles so no HDF5, or any thing for you just text.

MerlinSmiles commented 7 years ago

Im not saying just text?! I dont see the issue with HDF5, pretty standard thing? But i am very unsure about sql stuff for instance, I dont know much about it. And there are requirements for data archiving and stuff that we might not know about

jenshnielsen commented 7 years ago

outside the science world the use of SQL like databases is probably 1000 of times larger than HDF5 and sqlite even has build in support in python https://docs.python.org/3.5/library/sqlite3.html

MerlinSmiles commented 7 years ago

As I said I dont know much about it. Would just be a pitty to use something and in half a year your profs come and tell you no way.

jenshnielsen commented 7 years ago

Ok, makes sense I understood it as you were primarily concerned with the format going out of use in general and being to obscure

peendebak commented 7 years ago

@unga @merlinsmiles @jenshnielsen Both binary and text are fine with me. Sqlite is pretty standard, so quite future proof. However, i am not sure sqlite is really suitable for storing mixed type data (e.g. dicts containing strings, floats, numpy arrays, datasets and others dicts)

Both json and pickle do a good job of storing such structures, except for the dataset.