Closed mikelam14 closed 5 years ago
@mikelam14 can you impute/remove the missing value and try again?
The three dataframes (left, right, relation) has no missing values. Index of left and right, ids in relation are all integers. I group the data by mz.DataPack(relation=rel, left=req, right=adv)
. However pack.frame() would result in some nans. This affects unpack for sure. I suspect it also affects the reset_index in data_generator. If I change the indices to string, frame() would not work, saying vectors with NAN values cannot be used as index.
Could you provide a minimum working example that reproduces the bug so I can further investigate? Right now I suspect that you have mismatched indices somewhere but I can't be sure. Try calling pack
on your data without feeding indices (id_left
, id_right
) so pack
generates the correct ones for you.
I hope things are working well for you now. I’ll go ahead and close this issue, but I’m happy to continue further discussion whenever needed.
Describe the Question
I have prepared my own dataframes for left, right, relation. For left and right there are columns text{left/right}, length{left/right}, the index is id_{left/right}. All text is already converted into index and is truncated/padded to length=200. When I use the DataGenerator by
dev_generator = mz.DataGenerator( dev_processed, mode='pair', num_dup=1, num_neg=1, batch_size=20 )
, it returns error ofI have tried to print out the shapes and the content. I realized there is (nan, nan) pair from the batch_data_pack from
batch_data_pack = self._data_pack[indices]
. Maybe it is caused by reset_index? Thank you.Describe your attempts