define DatasetIterator.next to make it a real iterator

[yusefnapora]

This changes the DatasetIterator base class to define next and return self from the __iter__ method. The main "walker" generator function is now called generator, and the LocalFileIterator subclass now overrides generator instead of __iter__.

This gets rid of the confusing TypeError: LocalFileIterator object is not an iterator errors by making the DatasetIterator into an actual iterator.

[vyzo]

this doesn't support concurrent iterations, does it?

[yusefnapora]

Not directly, although you could call the generator method multiple times and iterate over that. But you could only iterate over the class instance itself once

[parkan]

@vyzo you mean to allow multiple threads to iterate over the same object? I honestly have no idea how thread safety works in python

[parkan]

I'll merge this for now so I can make my tests work

[vyzo]

no, I was talking about multiple iterations with the same object. It's fine to be iterable only once though, just pointing out the obvious :)