Open patricks-lab opened 1 year ago
Recently, I also found out that the error seems to originate from the utils.py (https://github.com/mboudiaf/pytorch-meta-dataset/blob/master/src/datasets/utils.py) file - sometimes, my program would hang after going into an infinite loop in cycle_ and eventually crashes with the above error.
So basically this is what happens:
First, in https://github.com/mboudiaf/pytorch-meta-dataset/blob/master/src/datasets/pipeline.py the code attempts to fetch a sample from a class, using sample_dic = next(self.class_datasets[class_id])
. But then this raises a TypeError: 'TFRecordDataset' object is not an iterator
exception, so it attempts to call cycle_()
on the class in the except clause.
def get_next(self, class_id):
try:
sample_dic = next(self.class_datasets[class_id])
except (StopIteration, TypeError) as e:
self.class_datasets[class_id] = cycle_(self.class_datasets[class_id])
sample_dic = next(self.class_datasets[class_id])
return sample_dic
But, in the function cycle_ in utils.py the code tries to yield the next sample in the iterator, and if not, it will attempt to reset the iterator. So the code looked like this:
def cycle_(iterable):
# Creating custom cycle since itertools.cycle attempts to save all outputs in order to
# re-cycle through them, creating amazing memory leak
iterator = iter(iterable)
while True:
try:
yield next(iterator)
except StopIteration:
iterator = iter(iterable)
The issue I think is happening is that there is some iteration where the class iterable is empty (something like iter([])
). I'm wondering if this may be due to a deformed/corrupted tfrecord (since I did specify a fixed 5-way, 10-shot task)? I'm thinking the class is empty since it's telling me that KeyError: "Key image doesn't exist (select from [])!"
, suggesting that the class we're trying to iterate from might be entirely empty.
As such I am wondering if I need to reinstall the files for meta-dataset? Or is there an issue with the code I'm running?
@mboudiaf I'm also interested in this
When sampling from the episodic loader, all usually goes fine until I get the following error:
Just for info - I used an older version of your repo (https://github.com/mboudiaf/pytorch-meta-dataset/tree/c6d6922003380342ab2e3509425d96307aa925c5). I am sampling from the episodic loader. I use
Do you know what may be causing the
KeyError: "Key image doesn't exist (select from [])!"
StopIteration error? For the above error, I am setting 5-way 15-shots for train and 5-way 5-shot for test/validation, and meta_batch_size 2 for train and 4 for test/val.EDIT: sometimes I'm also sampling from the episodic loader and encounter an infinite loop.
Thanks a lot in advance!