Open ideznaby opened 4 years ago
Hi @ideznaby , I've been experimenting with BRITS model recently. In my opinion data format output is incorrect in input_process.py. I've changed parse_rec to
def parse_rec(values, masks, evals, eval_masks, dir_):
deltas = parse_delta(masks, dir_)
ts_list = []
for r in range(values.shape[0]):
rec = {}
rec['values'] = np.nan_to_num(values[r]).tolist()
rec['masks'] = masks[r].astype('int32').tolist()
# imputation ground-truth
rec['evals'] = np.nan_to_num(evals[r]).tolist()
rec['eval_masks'] = eval_masks[r].astype('int32').tolist()
rec['deltas'] = deltas[r].tolist()
ts_list.append(rec)
return ts_list
Hi @Ubikas thank you for your answer, I tried this but I am still getting the same error, were you able to run the code on Physionet data?
Thanks
Hi @ideznaby ,
as far as I remember I was getting the same error with physionet data, but I adopted algorithm to my dataset and it has 19 attributes instead of 35. In my opinion to use json is not the best idea, I'd rather use numpy array with strict ts type order like [sample, ts_type, ts_length, attribute]. So in this case would be [2000,5,49,35]. Unfortunately I didn't have time to code it and as json format worked fine for me I left it as it was.
Instead of using usjon, just use python's original json API as follows in data_loader.py and input_process.py files:
# import ujson as json
import json
dumps()
and loads()
functions are the same. In input_process.py, define your own JSON encoder and dump rec
as follows:
class NpEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, np.integer):
return int(obj)
elif isinstance(obj, np.floating):
return float(obj)
elif isinstance(obj, np.ndarray):
return obj.tolist()
else:
return super(NpEncoder, self).default(obj)
rec = json.dumps(rec, cls=NpEncoder)
This effectively mitigated Maximum recursion level reached error in my code.
cls=NpEncoder
Thank you for your suggestions. It solved the problem of maximum recursion level issues. Btw have you got exact paper claimed results?
No, because PyTorch itself is non-deterministic. Please check: https://pytorch.org/docs/stable/notes/randomness.html
I am getting this error when I run input_process.py on physionet data, the error is happening on this line: rec = json.dumps(rec) do you have any solution for this? Thanks