DataSet class cannot be changed

[GoogleCodeExporter]

We cannot change the DataSet class, because the pickle files become invalid 
otherwise. Possible solutions

    define a "raw data Python format" from which it is fast to generated a DataSet and regenerate all pickle data.
        Advantage: a cleaner way to represent the data on disk 
    define a derived class DataSet2(DataSet)
        Advantage: data do not need to be regenerated
        Open question: can we pickle the super class DataSet without seeing the DataSet2 derived class? 

Original issue reported on code.google.com by dimo.bro...@inria.fr on 16 Jun 2014 at 1:02

[nikohansen]

Currently we are stuck to make DataSet a new style class. This breaks reading in pickle files. See also #19 #20 #45 #46

[nikohansen]

Here is a possible way out: we write a dump method that does something like

with os.open('dump_file') as f:
    f.write(repr(self.__dict__))

It may be a little more complicated, that is, we may need to copy and modify __dict__ and/or the result of repr for this specific need before to dump it. A quick try on a Python shell

>>> ds
DataSet(random_search on f6 2-D)
>>> dss = cma._BlancClass()
>>> len(dss.__dict__)
0
>>> dss.__dict__ = eval(repr(ds.__dict__))
NameError: name 'nan' is not defined
>>> nan = np.nan
>>> dss.__dict__ = eval(repr(ds.__dict__))
>>> len(ds.__dict__), len(dss.__dict__)
(20, 20)

suggests that only array and nan needs to be defined to succeed with eval(repr(self.__dict__)). However, repr writes floats apparently with limited precision. That is, the data are not exactly the same. AFAICS, this is not a major problem for our use case.

If we now create dss as DataSet from some arbitrary (but valid) input arguments first, we should be good.

Another entry point of investigation is JSON.

[dtusar]

We don't use pickle files any more.