Open fwbrasil opened 6 years ago
I wonder if the source you are dealing with has a bug with toIterator
? We assume we can call that again and again, but maybe this source has an issue there?
It seems to be a bug in cascading. TupleEntryChainIterator
should never throw if used correctly (hasNext
and then next
), which is the case.
I wonder if it is exhibited in cascading 2.7?
also: why did we not trigger it before, but now we do?
I'd love to find a repro of this issue.
I've investigated this issue a little more. The bug is not in TupleEntryChainIterator
, but in the underlying iterator impl HadoopTupleEntrySchemeIterator
. Its hasNext
returns true
initially but a second call to hasNext
returns false, even before next
is called.
@fwbrasil is this a race condition in Hadoop? we have seen a few of what looks like those.
One of our e2e tests fails when I try to use the the develop branch:
Considering that
Iterator.foreach
checks ifhasNext
before callingnext
, it seems thatTupleEntryChainIterator
enters a bad state wherecurrentIterator
points to an invalid position.I haven't been able to reproduce the cascading bug in isolation yet.
cc/ @johnynek