Chapter-10 Infinite iterator

[Roland-Szucs]

Information

The problem arises in chapter:

• [ ] Introduction
• [ ] Text Classification
• [ ] Transformer Anatomy
• [ ] Multilingual Named Entity Recognition
• [ ] Text Generation
• [ ] Summarization
• [ ] Question Answering
• [ ] Making Transformers Efficient in Production
• [ ] Dealing with Few to No Labels
• [x] Training Transformers from Scratch
• [ ] Future Directions

Describe the bug

class ConstantLengthDataset(IterableDataset) results infinite iteration. The reason for this is due to the exception handling part when we run out from the underlying dataset and catch the StopIteration exception. The code there:

To Reproduce

We do not need that as HF already created this code correctly just forgot to update this notebook. In this youtube video , the presented code is good. When the StopIteration is caught, the following good code is shown:

try:
    m=f"Fill buffer: {buffer_len}<{self.input_characters:.0f}"
    print(m)
    buffer.append(next(iterator)["content"])
    buffer_len += len(buffer[-1])
except StopIteration:
    more_examples = False
    break

try:
    m=f"Fill buffer: {buffer_len}<{self.input_characters:.0f}"
    print(m)
    buffer.append(next(iterator)["content"])
    buffer_len += len(buffer[-1])
except StopIteration:
    iterator = iter(self.dataset)

Expected behavior

Do not start the iteration if we just finished it otherwise it results infinit