search

chris-twiner / scalesXml

Alternate Scala XML library
77 stars 16 forks source link

Scalaz Iteratee / Enumeratee based on Id is doomed to fail #53

Closed chris-twiner closed 10 months ago

chris-twiner commented 11 months ago

Stackoverflows abound in actual usage for enumIterator or the custom version forcing Id. IO Trampoline's it seems. The M4 impl of iteratorEnumerator is also flawed now due to memory leaks. These are covered by various stack entries.

chris-twiner commented 11 months ago

moreover the use of iterator in the face of trampoline is also doomed (Id works fine). This is because enumerators are now baked into the iterator, the tests do:

val starter  = (resumableiteratee &= iteratorEnumerator(litr)).eval
..
foldM(starter)(
/// stuffs
  val next = (extractCont(starter)  &= iteratorEnumerator(litr)).eval
// more stuffs
)

There is no way to unpack the enumerator so calling with an xmliterator is deeply doomed. Most cases like normal onQNames shouldn't have an issue.

Given we want constant space we need to discard entries, this isn't even possible on the 500 long tests. It was observed that the evaluations are restarting, possibly something else is amiss. - commit 7c59489 (not pushed at time of commenting)

chris-twiner commented 10 months ago

as per the workaround, wrapping iterators in either WeakStream or Ephemeral (obvs preferred) and a leaky enumerator allows capturing progress. The tests work, the code is trampolined, and it's not totally hideous if needed.

Both should free up memory but still needs checking for space