elypma
commented
5 years ago After some debugging it appeared that both JsonTarget and PickleTarget have to be adopted for the use of pathlib - and then the opening of files for reading and writing is done using the member function open of the path variable.
class JsonTarget(DataTarget):
"""
Saves to json, loads to dict
"""
def load(self, cached=False, **kwargs):
"""
Load from json to dict
Args:
cached (bool): keep data cached in memory
**kwargs: arguments to pass to json.load
Returns: dict
"""
def read_json(path, **opts):
with path.open('r') as fhandle:
df = json.load(fhandle)
return df['data']
return super().load(read_json, cached, **kwargs)
def save(self, dict_, **kwargs):
"""
Save dict to json
Args:
dict_ (dict): python dict
kwargs : additional arguments to pass to json.dump
Returns: filename
"""
def write_json(path, _dict_, **opts):
with path.open('w') as fhandle:
json.dump(_dict_, fhandle, **opts)
(self.path).parent.mkdir(parents=True, exist_ok=True)
opts = {**{'indent':4},**kwargs}
write_json(self.path, {'data':dict_}, **opts)
return self.path
class PickleTarget(DataTarget):
"""
Saves to pickle, loads to python obj
"""
def load(self, cached=False, **kwargs):
"""
Load from pickle to obj
Args:
cached (bool): keep data cached in memory
**kwargs: arguments to pass to pickle.load
Returns: dict
"""
return super().load(lambda x: pickle.load(x.open("rb")), cached, **kwargs)
def save(self, obj, **kwargs):
"""
Save obj to pickle
Args:
obj (obj): python object
kwargs : additional arguments to pass to pickle.dump
Returns: filename
"""
(self.path).parent.mkdir(parents=True, exist_ok=True)
pickle.dump(obj, self.path.open("wb"), **kwargs)
return self.path
I am trying to run the d6tflow example (unmodified):
but this fails at the last line: self.save(model):
For some reason it cannot create the file. The 'data' directory is created, it contains 2 subdirectories: TaskGetData (containing a pq file as expected) and a TaskTrain directory (which is empty).
I am running on windows 10, and I am using python 3.5.
Any idea what might be wrong?