The trouble with XPath‘s fn:fold-right. A fix and Proposal for fn:fold-lazy

[dnovatchev]

The trouble with XPath‘s fn:fold-right.
Laziness in XPath.

This article discusses the standard XPath 3.1 function fn:fold-right, its definition in the official Spec, its lack of apparent use-cases and its utter failure to reproduce the (lazy) behavior of Haskell’s foldr , which is presumed to be the motivation behind fn:fold-right.
The 2nd part of the article introduces the implementation of short-circuiting and generators, which together unprecedentedly provide laziness in XPath. Based on these, a new XPath function: fn:fold-lazy is implemented, that utilizes laziness, similar to Haskell’s foldr. This behavior is demonstrated in specific examples

Introduction

Higher order functions were introduced into XPath starting with version 3.0 in 2014 and later in version 3.1 in 2017.
The definition of the standard function fn:fold-right closely mimics that of Haskell’s foldr, and anyone acquainted with foldr can be left with the impression that fn:fold-right would have identical behavior (and hence use-cases) as Haskell’s foldr.

Unfortunately, there is a critical difference between the definitions of these two functions. Whereas the definition of foldr explicitly defines its behavior when provided with a function, lazy in its 1st argument – from Haskell’s definition of foldr:

“… Note that since the head of the resulting expression is produced by an application of the operator to the first element of the list, given an operator lazy in its right argument, foldr can produce a terminating expression from an unbounded list.”

The XPath definition of fn:fold-right does not mention any laziness.

There is no official concept of “laziness” in XPath, thus fn:fold-right doesn’t cover some of the most important use-cases of Haskell’s foldr , which can successfully produce a result when passed an infinite (or having unlimited length) list.

This in fact makes fn:fold-right almost useless, and explains why even some of the members of the XPath 3.1 WG have stated on occasions that they do not see why the function was introduced.

fn:fold-right gone wrong – example

This Haskell code:

foldr (\x y -> (if x == 0 then 0 else x*y)) 1 (map (\x -> x - 15) [1 ..1000000])

foldr (\x y -> (if x == 0 then 0 else x*y)) 1 (map (\x -> x - 15) [1 ..10000000])

foldr (\x y -> (if x == 0 then 0 else x*y)) 1 (map (\x -> x - 15) [1 ..])

produces the product of all numbers in the following list, respectively:

[-14, -13, -12, -11, -10, -9, -8, -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, …, 999985]

[-14, -13, -12, -11, -10, -9, -8, -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, …, 9999985]

[-14, -13, -12, -11, -10, -9, -8, -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, …, ] -- up to infinity.

Because all these 3 lists contain a zero as their 15th item, the expected result is 0 when evaluating any of these 3 expressions – even in the last case where the provided as argument list is infinite. And this is indeed what happens:

Not only Haskell produces the correct result in all cases, but regardless of the list’s length, the result is produced instantaneously!

Now, let us evaluate this equivalent XPath expression with BaseX:

let $product := function($input as array(xs:integer)) as xs:integer
                         { 
                           array:fold-right($input, 1, function($x as xs:integer, $y as xs:integer) as  xs:integer 
                                                               {if($x eq 0) then 0 else $x * $y}) 
                         },
    $ar := array { (1 to 36) ! function($x as xs:integer) as xs:integer {$x -15}(.)}
  return
     $product($ar)

Here we are passing a list containing just 36 integers. The result is quite unexpected and spectacular:

Here is what happens:

1. Even though when processing the 15th integer in the array the result is 0, the XPath processor continues to evaluate the RHS (right-hand side) until the last member of the array (36).

2. On “its way back” the XPath processor multiplies: (36*35*34*33*32* …*6*5*4)*3, and the result of the right-most multiplication is bigger than the maximum integer (or decimal) that this XPath processor supports.

3. C r r r a s s h … As seen in the screenshot above.

The root cause for this unfortunate behavior is that the XPath processor doesn’t support short-circuiting and laziness. And thus, fn:fold-right is useless even in the normal/trivial case of a collection (array) with only 36 members. Not to speak of collections containing millions of members, or even infinite ones…

Let us see what happens when evaluating similar expressions with another XPath processor: Saxon.

Saxon seems to produce the correct result, however it takes exponentially longer times when the length of the passed array is increased, leading to this one:

It took 261 seconds for the evaluation to be done, but accessing the 15th member of the array and short-circuiting to 0 should be almost instantaneous…
So what happens in this case? The difference between BaseX and Saxon is that Saxon implements a “Big Integer” and thus can multiply almost 1 000 000 integers without getting a value that cannot be handled… But doing almost 1M multiplications of big integers obviously takes time …

What is common in these two examples? Obviously, neither BaseX nor Saxon detects and performs short-circuiting. Why is this? What is the reason for this?

I asked a developer of BaseX if I could submit a bug about this behavior. His answer was shockingly unexpected: “This is not a bug, because no requirement in the Specification has been violated”.

Thus, the main cause of the common behavior of both XPath processors to handle the evaluation of these examples, is the specification of the function, which blatantly allows such crap to happen.

Now that we see this, let us try to provide the wanted, useful behavior writing our own function.

The fix: Step 1 – fn:fold-right in pure XPath

Before going in depth with our pure XPath solution, we need as a base a pure-XPath implementation of fn:fold-right .

 let $fold-right-inner := function ($seq as item()*,
                                    $zero as item()*,
                                    $f as function(item(), item()*) as item()* ,
                                    $self as function(*)
                                   ) as item()*
{
  if(empty($seq)) then $zero
    else
      $f(head($seq), $self(tail($seq), $zero, $f, $self))
},

    $fold-right := function ($seq as item()*,
                             $zero as item()*,
                             $f as function(item(), item()*) as item()* 
                            ) as item()*
{
  $fold-right-inner($seq, $zero, $f, $fold-right-inner)
},

   $fAdd := function($x, $y)  {$x + $y},
   $fMult  := function($x, $y)  {$x * $y}

   return
     $fold-right((1 to 6) ! function($x){$x - 3}(.), 1, $fMult)

When we evaluate the above with any of the two XPath processors, the correct result is produced:

0

And we certainly do have exactly the same problems as the provided built-in fn:fold-right with a similar example:

The fix: Step 2 – $fold-right-sc detecting and performing short-circuiting

Now that we have $fold-right as a base, let us add code to it so that it will detect and perform short-circuiting. We will implement a function similar to $fold-right but having this signature:

    $fold-right-sc := function ($seq as item()*,
                                $zero as item()*,
                                $f as function(item(), item()*) as item()*,
                                $fGetPartial as function(*)
                               ) as item()*

The last of the function’s parameters $fGetPartial returns a new function that is the partial application of $f, when its 1st argument is set to the current member of the input sequence $seq. The idea is that whenever short-circuiting is possible, $fGetPartial returns not a function having one argument (arity 1), but a constant – a function with 0 arguments (arity 0).

If the arity of the so produced partial application is 0, then our code will immediately return with the value $f($currentItem).

Here is the complete code of $fold-right-sc:

 let $fold-right-sc-inner := function ($seq as item()*,
                                       $zero as item()*,
                                       $f as function(item(), item()*) as item()*,
                                       $fGetPartial as function(*),
                                       $self as function(*)
                                      ) as item()*
{
  if(empty($seq)) then $zero
    else
      if(function-arity($fGetPartial(head($seq), $zero)) eq 0)
        then $fGetPartial(head($seq), $zero) ()
        else $f(head($seq), $self(tail($seq), $zero, $f, $fGetPartial, $self))
},

    $fold-right-sc := function ($seq as item()*,
                                $zero as item()*,
                                $f as function(item(), item()*) as item()*,
                                $fGetPartial as function(*)
                               ) as item()*
{
  $fold-right-sc-inner($seq, $zero, $f, $fGetPartial, $fold-right-sc-inner)
},

   $fAdd := function($x, $y)  {$x + $y},
   $fMult  := function($x, $y)  {if($x eq 0) then 0 else $x * $y},
   $fMultGetPartial := function($x, $y)
   {
     if($x eq 0)
       then function() {0}
       else function($z) {$x * $z}
   }

   return
     $fold-right-sc((1 to 1000000) ! function($x){$x - 3}(.), 1, $fMult, $fMultGetPartial)

Do note:

1. If the current item (the head of the sequence) is 0, then $fMultGetPartial returns a function with 0 arguments (constant) that produces 0.

2. $fold-right-sc (inner) treats differently a partial application of arity 0 from a partial application with arity 1. In the former case it simply produces the expected constant value without recursing further. Here is the relevant code fragment

  if(empty($seq)) then $zero
    else
      if(function-arity($fGetPartial(head($seq), $zero)) eq 0)
        then $fGetPartial(head($seq), $zero) ()
        else $f(head($seq), $self(tail($seq), $zero, $f, $fGetPartial, $self))

And now BaseX has no problems with the evaluation, even though the input sequence is of size 1M. The complete evaluation takes just a fraction of a millisecond (0.04 ms):

With Saxon things are not so good. Even though Saxon produces the correct result, evaluating the expression with an input sequence of size 1M takes 0.5 seconds (half a second), and evaluating the expression with an input sequence of 10M takes 5 seconds (10 times as long):

What is happening?

Even though Saxon performs much faster than the previous 261 seconds, due to detecting the short-circuiting possibility and performing the short-circuit, Saxon still processes all 10M items when evaluating this subexpression (which obviously the more optimized BaseX doesn’t do in advance):

(1 to 10000000) ! function($x){$x - 3}(.)

Therefore, we have one remaining problem: How to prevent long sequences (or arrays) from being fully materialized before starting the evaluation of $fold-right-sc ?

The fix: Step 3 – replacing collections with generators

Generators are well known and provided out of the box in many programming languages. Per Wikipedia:

“In computer science, a generator is a routine that can be used to control the iteration behaviour of a loop. All generators are also iterators.[1] A generator is very similar to a function that returns an array, in that a generator has parameters, can be called, and generates a sequence of values. However, instead of building an array containing all the values and returning them all at once, a generator yields the values one at a time, which requires less memory and allows the caller to get started processing the first few values immediately. In short, a generator looks like a function but behaves like an iterator.”

A full-fledged generator (such as implemented in C#) is an instance of a Finite State Machine(FSM), and implementing it in full generality goes beyond the topic and goals of this article. Expect another article soon that will provide this.

Here we will implement a simple kind of generator, that when passed an integer index $N, produces the $Nth item of a specific sequence. Although this is probably the simplest form of a generator, it can be useful in many cases and is a good illustrative solution to our current problem. The whole approach of replacing “something” with a function that must be called to produce this “something” is known as “lifting”

First, we will add to our $fold-right just the use of generators, without the detection and performing of short-circuiting:

let $fold-right-lifted-inner := function ($seqGen as function(xs:integer) as array(*),
                                    $index as xs:integer,
                                    $zero as item()*,
                                    $f as function(item(), item()*) as item()* ,
                                    $self as function(*)
                                   ) as item()*
                                {
                                  let $nextSeqResult := $seqGen($index),
                                      $isEndOfSeq :=  $nextSeqResult(1),
                                      $seqItem := $nextSeqResult(2)
                                    return
                                      if($isEndOfSeq) then $zero
                                        else
                                          $f($seqItem, $self($seqGen, $index+1, $zero, $f, $self))
                                },

    $fold-right-lifted := function ($seqGen as function(xs:integer) as array(*),
                                    $zero as item()*,
                                    $f as function(item(), item()*) as item()* 
                                  ) as item()*
                                  {
                                    $fold-right-lifted-inner($seqGen, 1, $zero, $f, $fold-right-lifted-inner)
                                  },

   $NaN := xs:double('NaN'),

   $fSeq1ToN := function($ind as xs:integer, $indStart as xs:integer, $indEnd as xs:integer) as array(*)
                {
                  if($ind lt  $indStart or $ind gt $indEnd)
                    then  array{true(), $NaN}
                    else array{false(), $ind}
                },
   $fSeq-1-6 := $fSeq1ToN(?, 1, 6),

   $fAdd := function($x, $y)  {$x + $y},
   $fMult  := function($x, $y)  {$x * $y}

   return
     $fold-right-lifted($fSeq-1-6, 1, $fMult) 

Here we see an example of a simple generator – the function $fSeq1ToN.

This function returns an array with two members: a Boolean, which if true() indicates the end of the sequence, and the 2nd member is the current head of the simulated sequence.
The generator has two other parameters which are the values (inclusive) for the start-index and the end-index. Whenever the passed value of $ind is outside of this specified range, $fSeq1ToN returns a result array with its first member set to true() (the 2nd member of the result must be ignored in this case), which indicates end-of sequence.
Otherwise it returns array{false(), $ind} . It is the responsibility of the caller to stop calling the generator:

   $fSeq1ToN := function($ind as xs:integer, $indStart as xs:integer, $indEnd as xs:integer) as array(*)
                {
                  if($ind lt  $indStart or $ind gt $indEnd)
                    then  array{true(), $NaN}
                    else array{false(), $ind}
                }

Evaluating the complete XPath expression above produces the correct result both in BaseX and in Saxon: the product of the integers 1 to 6:

Now that we have successfully implemented the last missing piece of our complete solution, let us put everything together:

The fix: Step 4 – putting it all together

Finally we can replace the input sequence in \$fold-right-sc with a generator:

let $fold-right-sc-lifted-inner := function ($seqGen as function(xs:integer) as array(*),
                                    $index as xs:integer,
                                    $zero as item()*,
                                    $f as function(item(), item()*) as item()* ,
                                    $fGetPartial as function(*),
                                    $self as function(*)
                                   ) as item()*
                                {
                                  let $nextSeqResult := $seqGen($index),
                                      $isEndOfSeq :=  $nextSeqResult(1),
                                      $seqItem := $nextSeqResult(2)
                                    return
                                      if($isEndOfSeq) then $zero
                                        else
                                          if(function-arity($fGetPartial($seqItem, $zero)) eq 0)
                                            then $fGetPartial($seqItem, $zero) ()
                                            else $f($seqItem, $self($seqGen, $index+1, $zero, $f, $fGetPartial, $self))
                                },

    $fold-right-sc-lifted := function ($seqGen as function(xs:integer) as array(*),
                                       $zero as item()*,
                                       $f as function(item(), item()*) as item()*,
                                       $fGetPartial as function(*) 
                                      ) as item()*
                                      {
                                         $fold-right-sc-lifted-inner($seqGen, 1, $zero, $f, $fGetPartial, $fold-right-sc-lifted-inner)
                                      },

   $NaN := xs:double('NaN'),

   $fSeq1ToN := function($ind as xs:integer, $indStart as xs:integer, $indEnd as xs:integer) as array(*)
                {
                  if($ind lt  $indStart or $ind gt $indEnd)
                    then  array{true(), $NaN}
                    else array{false(), $ind}
                },
   $fSeq-1-6 := $fSeq1ToN(?, 1, 6),
   $fSeq-1-1M := $fSeq1ToN(?, 1, 1000000),
   $fSeq-1-1M-minus-3 := function($n as xs:integer)
   {
     array{$fSeq-1-1M($n)(1), $fSeq-1-1M($n)(2) -3}
   },

   $fAdd := function($x, $y)  {$x + $y},
   $fMult  := function($x, $y)  {$x * $y},
   $fMultGetPartial := function($x, $y)
   {
     if($x eq 0)
       then function() {0}
       else function($z) {$x * $z}
   }

   return
     $fold-right-sc-lifted($fSeq-1-1M-minus-3, 1, $fMult, $fMultGetPartial) 

Now this expression (and even one involving a sequence of 10M items take 0 seconds to be evaluated in both BaseX and Saxon, producing the correct result 0:

Summary

This article demonstrated the problems inherent to the standard XPath fn:fold-right and correctly determined the root causes for these problems: no short-circuiting and no collection generators.

Then a step-by-step solution was built that shows how to implement lazy evaluation in XPath based on short-circuiting and collection generators. This fixed the error raised by BaseX and dramatically reduced the evaluation time of Saxon from 261 seconds to 0 seconds.

The new function produced can be called $fold-lazy and is a good candidate for inclusion in the XPath 4.0 standard functions.

A complete design and implementation of a general collection-generator will be published in a separate article.

[michaelhkay]

Thanks for this very thorough analysis. It will take me some time to study it and do it justice. But I'm sure you are right that fold-right was added to the spec without any clear understanding of why it was needed and what use cases it was intended to serve, and therefore what performance profile might be expected of it.

One question: the original formulation uses the function:

function($x as xs:integer, $y as xs:integer) as  xs:integer 
     {if($x eq 0) then 0 else $x * $y}

which is obviously equivalent to

function($x as xs:integer, $y as xs:integer) as  xs:integer 
     {$x * $y}

You seem to be expecting that the fold-right evaluation will be able to do an early exit if the first formulation is used, but (presumably) not if the second one is used. I'm wondering why? Surely the essential ingredient is that in both cases the processor has to be able to detect that the function is idempotent when $x eq 0? Your expansion using $fGetPartial seems to be there largely in order to supply this missing knowledge.

A second question: why use fold-right for this problem rather than fold-left? How would the analysis differ if you used fold-left? It seems to me that in both cases, the short-cutting is possible if the processor can recognize the idempotence of zero, and impossible otherwise.

[ChristianGruen]

Thanks, Dimitre, for the analysis.

Many programming languages restrict themselves to a variant of fold-left (usually called reduce) as that covers the most common requirement (iterative processing of data in a functional language). Both fold-left and fold-right can be implemented differently: iteratively (as a classical for loop) or recursively (i.e., equivalent to the equivalent XQuery solutions that are presented in the spec).

Both approaches have advantages and drawbacks. The implementations I have tried (Saxon, eXist-db, Zorba, XmlPrime, BaseX) seem to have chosen an iterative implementation for both functions (as far as I can judge). I can only guess why, but our motivation for not choosing the recursive approach was to safeguard users against stack overflow errors caused by code that cannot be optimized for tail recursion. A simple example (similar to the one in the main comment above):

declare function local:fold-right($seq, $data, $f) {
  if (empty($seq)) then $data else $f(head($seq), local:fold-right(tail($seq), $data, $f))
};
(: local: can be dropped to use the buit-in function :)
local:fold-right(1 to 10000, 0, op('+'))

@dnovatchev: Looking forward to how fn:fold-lazy would look like.

[ChristianGruen]

PS: As Dimitre has pointed out, BaseX does not support integers larger than 2^64. The easiest solution to circumvent the restriction, and avoid the error message, is to switch to xs:decimal.

[michaelhkay]

Presumably if you want to be sure of early exit without relying on the processor detecting idempotence, you can write this as

iterate-while($seq,
              fn($seq){exists(tail($seq))},
              fn($seq){if (head($seq) eq 0) then 0 else ($seq[1]*$seq[2], subsequence($seq, 3))

[ChristianGruen]

Regarding fn:iterate-while, for an early exit, the condition should probably be specified in the first function:

iterate-while(
  $input,
  fn($data) { head($data) != 0 and exists(tail($data)) },
  fn($data) { $data[1] * $data[2], subsequence($data, 3) }
)[1]

Another solution that separates the processed data and the input sequence:

let $result := iterate-while(
  [ 1, $input ],
  fn($data) { $data(1) != 0 and exists($data(2)) },
  fn($data) { [ $data(1) * head($data(2)), tail($data(2)) ] }
)
return $result(1)

…and another one with the positional argument proposed in #516:

iterate-while(
  $input,
  fn($result, $pos) { $result != 0 and $pos <= count($input) },
  fn($result, $pos) { $result * $input[$pos] }
)

[dnovatchev]

One question: the original formulation uses the function:

function($x as xs:integer, $y as xs:integer) as  xs:integer 
     {if($x eq 0) then 0 else $x * $y}

which is obviously equivalent to

function($x as xs:integer, $y as xs:integer) as  xs:integer 
     {$x * $y}

You seem to be expecting that the fold-right evaluation will be able to do an early exit if the first formulation is used, but (presumably) not if the second one is used. I'm wondering why?

The reason is that even in Haskell this expression doesn't cause short circuiting:

foldr (*) 1 [-15..1000000]

and the evaluation takes excessive time and ends in Error (insufficient memory):

The processor is not a position to recognize (prove) that a given value will cause short-circuiting for any operation that is provided as parameter.

The known (and verified) example that works with Haskell's foldr is this:

foldr (\x y -> (if x == 0 then 0 else x*y)) 1 (map (\x -> x - 15) [1 ..])

Surely the essential ingredient is that in both cases the processor has to be able to detect that the function is idempotent when $x eq 0? Your expansion using $fGetPartial seems to be there largely in order to supply this missing knowledge.

Yes, because the processor cannot be relied to do such detection reliably even for a small, fixed, finite set of operations, we are giving it hints for any such value - known only to us.

Also, imagine that the function $f is "external", that is its code is not available to the processor and the processor regards it as a black box. In any such case even the smartest processor needs short-circuiting hints.

A second question: why use fold-right for this problem rather than fold-left? How would the analysis differ if you used fold-left? It seems to me that in both cases, the short-cutting is possible if the processor can recognize the idempotence of zero, and impossible otherwise.

This is easy to answer: even though it is possible to detect a short-circuiting value while evaluating fold-left , due to the function definition, it is impossible to perform an immediate exit and all previous N - 1 enclosing applications of fold-left need to be executed. This is because in Haskell the evaluation is always left to right, and thus short-circuiting is detected in the 1st argument of an operation, but fold-left will need to check for a short-circuiting of this operation's 2nd argument.

Extensive explanation can be found in this Haskell official Wiki:

https://wiki.haskell.org/Foldr_Foldl_Foldl'

[dnovatchev]

Regarding fn:iterate-while, for an early exit, the condition should probably be specified in the first function:

@ChristianGruen , @michaelhkay Yes, a variation of fn:iterate-while can be made "lazy" if it accepts as input not a sequence but a generator.

Using a generator to provide the members of a possibly huge or infinite collection can benefit not only right-folding but many other type of operations that need to traverse over a collection.

…and another one with the positional argument proposed in #516:

iterate-while(
  $input,
  fn($result, $pos) { $result != 0 and $pos <= count($input) },
  fn($result, $pos) { $result * $input[$pos] }
)

This one will fail with huge sequences, as it uses count($input) which necessitates traversing the whole input and is prohibitive for huge or infinite inputs.

[ChristianGruen]

@ChristianGruen , @michaelhkay Yes, a variation of fn:iterate-while can be made "lazy" if it accepts as input not a sequence but a generator.

Laziness is a different topic. I think this (or at least my intention) was to demonstrate how the evaluation can be stopped once a condition is true, no matter how large the input is.

This one will fail with huge sequences, as it uses count($input) which necessitates traversing the whole input and is prohibitive for huge or infinite inputs.

True, counting infinite input wouldn’t be possible; but I wonder how you’d generate such input at all in XPath/XQuery. If $input is of limited size, it’s trivial to compute the length even if it’s huge.

[ChristianGruen]

even though it is possible to detect a short-circuiting value while evaluating fold-left , due to the function definition, it is impossible to perform an immediate exit and all previous N - 1 enclosing applications of fold-left need to be executed.

I think it would easily be possible with an iterative evaluation of the fold functions…

Value result = zero
for(Item item : input) {
  if short-circuit check successful: break
  result = action(result, item)
}

…provided that we understand how we can do the detection.

[dnovatchev]

I think it would easily be possible with an iterative evaluation of the fold functions…

Value result = zero
for(Item item : input) {
  if short-circuit check successful: break
  result = action(result, item)
}

…provided that we understand how we can do the detection.

Sorry, this code is not written in XPath ... 😄

And also (correct me if I am wrong) iterative evaluation supposes that you have traversed the complete collection and are starting the multiplications from the end of the collection. This is impossible with infinite collections and would take prohibitively long with huge collections.

[ChristianGruen]

@dnovatchev Thanks again for mentioning…

The known (and verified) example that works with Haskell's foldr is this: foldr (\x y -> (if x == 0 then 0 else x*y)) 1 (map (\x -> x - 15) [1 ..])

Do you know more about this pattern/has it been formalized somewhere?

Sorry, this code is not written in XPath ... 😄

It isn’t, and it wasn’t supposed to be. The existing implementations of fn:fold-left of fn:fold-right seem to be iterative, and I wanted to show how straightforward it would be to support short-circuiting in these implementations without recursive code… provided that the function body of the action matches a known pattern.

To short-circuit if x == 0 then 0 else x*y iteratively, an XQuery optimizer could check…

• whether the function body is an if/then/else expression,
• whether the condition does not rely on y
• whether the then branch returns x (or a value equal to x).

If these conditions apply, the underlying implementation (…not the XPath code)…

result = zero
for(item : input) {
  result = action(result, item)
}

…could be replaced by an implementation that invokes the condition and the else branch separately:

result = zero
for(item : input) {
  if condition(result): break
  else: result = else-branch(result, item)
}

Obviously, this would need to be generalized in practice to support different kinds of guarded expressions.

And also (correct me if I am wrong) iterative evaluation supposes that you have traversed the complete collection and are starting the multiplications from the end of the collection.

That’s not required (at least in our processor). For simplicity, I’ve used item : input in the code above, but it can be an iterator, starting with the first item (for fold-left) or the last item (for fold-right).

[dnovatchev]

And also (correct me if I am wrong) iterative evaluation supposes that you have traversed the complete collection and are starting the multiplications from the end of the collection.

That’s not required (at least in our processor). For simplicity, I’ve used item : input in the code above, but it can be an iterator, starting with the first item (for fold-left) or the last item (for fold-right).

This is exactly what I was stating: How can the processor know "the last item" of a huge or infinite list?

This iterative approach for fold-right requires traversing the whole input (to get to the last item) before even starting the processing.

In a right fold the first result is only produced upon reaching the last member of the input collection, which happens never for infinite input, and "years away" for lists huge enough.

[ChristianGruen]

This is exactly what I was stating: How can the processor know "the last item" of a huge or infinite list?

Give me an example: How would you generate an infinite list with XPath?

[dnovatchev]

This is exactly what I was stating: How can the processor know "the last item" of a huge or infinite list?

Give me an example: How would you generate an infinite list with XPath?

Simply creating a generator with upper-index limit set to xs:double('INF')

This is one more unprecedented feature and use for XPath generators -- and is just easy and natural 😄

[ChristianGruen]

Simply creating a generator with upper-index limit set to xs:double('INF')

Simple enough to write it down? ;)

[dnovatchev]

Simply creating a generator with upper-index limit set to xs:double('INF')

Simple enough to write it down? ;)

Sure, let me find a good example.

here is the example - in a comment down this thread 😄

[michaelhkay]

This is exactly what I was stating: How can the processor know "the last item" of a huge or infinite list?

The same way as it knows the first item: because the list is implemented using a data structure that gives direct access to the last item. Unlike Haskell, there is nothing in XPath that says list implementation should be biassed towards left-to-right traversal. With some sequences, for example preceding::*, right-to-left traversal is much easier than left-to-right. Other sequences, such as (1 to 1_000_000) or (child::*) can be processed equally easily in either direction; and child::*[last()] in a typical tree implementation can be evaluated without accessing any children other than the last.

[michaelhkay]

Although details of the Saxon implementation are not really pertinent to the spec, for the benefit of anyone who wants to get the best performance out of Saxon, I would point out that at present we never do lazy evaluation of arrays -- only of sequences. There's no intrinsic reason for this, other than the fact that our priorities for performance work tend to be driven by customer use cases, and use of arrays for most of our users is still something of a novelty.

The story as presented by Dimitre seems to switch from arrays to sequences halfway through without warning.

With SaxonJ-EE 12.3 I get the following timings for a sequence of 100K / 200K integers starting at -15:

• array:fold-right: 2.95s / 10.45s
• array:fold-left: 14.14ms / 24.77ms
• fn:fold-right: 2.86s / 10.48s
• fn:fold-left: 13.12ms / 24.66ms

So it seems there is very little difference between sequences and arrays (and therefore between lazy and eager evaluation), but a big difference between fold-left and fold-right (fold-left is clearly linear, fold-right is clearly quadratic -- fortunately not exponential as suggested above!)

Of course none of these is doing an early exit, because the fixpoint zero is not recognised.

I need to explore why Saxon's implementation of fold-right is quadratic, but that investigation is not really relevant to this issue. Those who are interested may follow the trail at https://saxonica.plan.io/issues/6192. Intrinsically I see no reason why it should not be possible to implement fold-right with linear performance without making any changes to the spec. Achieving early exit when a fixpoint is reached is of course another matter entirely.

UPDATE:

It turns out that Saxon's fold-right isn't intrinsically quadratic in performance; it just appears so in this example, because of the increasing size of the integers being multiplied (which do indeed increase exponentially with the length of the input sequence).

When we use a slightly different function -- one which multiplies successive doubles in the range 0..1 -- we find that there is very little difference between fold-left and fold-right, or between sequences and arrays. In all cases the performance is O(n).

The only thing therefore that will improve the performance of the original query is detection of the fixpoint. That's something that could be done by the optimizer in a few special cases, though I suspect that they are sufficiently rare that it's not worth the effort.

Lazy evaluation doesn't come into it. Saxon is doing lazy evaluation for sequences and not for arrays, and it seems to make very little difference. The only thing that will make a difference is early exit when the fixpoint is detected. I think the number of cases where that can be done automatically by an optimiser is probably rather small.

[ChristianGruen]

For those who are interested, I've just added the proposed shortcut solution in BaseX. With the latest snapshot, you can evaluate the following query in a few milliseconds:

let $seq := (1 to 1000000000000000000)
return (
  fold-left($seq, 1, fn($r, $i) { true() }),
  fold-left($seq, 1, fn($r, $i) { $r }),
  fold-left($seq, 1, fn($r, $i) { if($r > 100) then $r else $r + $i }),
  fold-left($seq, 1, fn($r, $i) { if($r < 100) then $r + $i else $r }),
  fold-right($seq, 1, fn($i, $r) { true() }),
  fold-right($seq, 1, fn($i, $r) { $r }),
  fold-right($seq, 1, fn($i, $r) { if($r > 100) then $r else $r + $i }),
  fold-right($seq, 1, fn($i, $r) { if($r < 100) then $r + $i else $r })  
)

[ChristianGruen]

Here’s another example for input that’s iteratively evaluated (as only the last item is requested, error() will never be called):

foot((1, error(), (1 to 1234567890) ! (. + 1)))

And a slightly more complex example for fold-right and early-exit processing:

fold-right(
  reverse(1 to 10000000000000) ! (. * 2) ! element _ { . },
  1,
  fn($i, $r) { if($r * $r <= 10000) then $r + $i else $r }
)

As usual, it’ll be up to the implementation to choose the most appropriate evaluation strategy.

[michaelhkay]

Generators do seem to be a potentially very powerful tool, though probably only one for expert users. I can see this being made available as a function

fn:generate-sequence($len as xs:nonNegativeInteger, $get as function(xs:positiveInteger) as item()) where the result is a sequence of length $len, whose $Nth item can be determined by calling $get($N).

Plus an equivalent for arrays.

I'm having a little trouble finding concrete use cases, however. One might be for sparse arrays.

There's a relationship here with virtual maps: see issue #105.

[dnovatchev]

This is exactly what I was stating: How can the processor know "the last item" of a huge or infinite list?

The same way as it knows the first item: because the list is implemented using a data structure that gives direct access to the last item. Unlike Haskell, there is nothing in XPath that says list implementation should be biassed towards left-to-right traversal. With some sequences, for example preceding::*, right-to-left traversal is much easier than left-to-right. Other sequences, such as (1 to 1_000_000) or (child::*) can be processed equally easily in either direction; and child::*[last()] in a typical tree implementation can be evaluated without accessing any children other than the last.

As before: This is not applicable to an infinite collection.

Also, storing a huge collection in-memory is a huge (repetition 😄 ) waste of resources.

[dnovatchev]

The story as presented by Dimitre seems to switch from arrays to sequences halfway through without warning.

Yes, because the problem and the solution equally apply to both.

I didn't want to make/replicate the code twice (and make it twice as much difficult to read) just for minor replacements of arrays <---> sequences

[dnovatchev]

Generators do seem to be a potentially very powerful tool, though probably only one for expert users. I can see this being made available as a function

fn:generate-sequence($len as xs:nonNegativeInteger, $get as function(xs:positiveInteger) as item()) where the result is a sequence of length $len, whose $Nth item can be determined by calling $get($N).

Plus an equivalent for arrays.

I'm having a little trouble finding concrete use cases, however. One might be for sparse arrays.

There's a relationship here with virtual maps: see issue #105.

The proposed function is very limited and only scratches on the surface of the topic about generators:

1. It doesn't support generating infinite collections

2. It only supports the most primitive type of generator that can provide the next result based only on the current index

A generator in its general form is an FSM (Finate State Machine).

As the name FSM suggests, the generator can have many states, and in order to continue from the current point, and to produce the next result, it has to access (to be passed by the caller as an argument) its complete last state. For example, a generator for the Fibonacci sequence typically has a state that includes its last 2 results. In practice there may be generators that have a much more complex state. The state of a generator is an object (map in XPath parlance) that has some client-visible members, such as latest-result, and move-next, and any number of "internal" members that are of interest only to the generator itself. Probably this could well be modelled using the XPath 4.0 record type.

The "next state" returned by a generator is suitable to store as part of the accumulator argument of a fold function, so that the end user doesn't have to take care of receiving, maintaining the state, and passing it in a next call to the generator.

As said previously, I will describe such generators with code examples in another issue.

[dnovatchev]

So it seems there is very little difference between sequences and arrays (and therefore between lazy and eager evaluation), but a big difference between fold-left and fold-right (fold-left is clearly linear, fold-right is clearly quadratic -- fortunately not exponential as suggested above!)

Actually, what is said to be in Saxon "lazy"(I very much doubt that this is really lazy) evaluation with sequences behaves significantly worse than Saxon's "eager" evaluation with arrays:

Similarly to the code in the opening comment, but now using just fn:fold-right (instead of array:fold-right) and passing to it a sequence, takes 275 seconds vs. the 260 seconds seen for Saxon's "eager" evaluation with arrays.:

let $product := function($input as xs:integer*) as xs:integer
                {  fold-right($input, 1, function($x as xs:integer, $y as xs:integer) as xs:integer {if($x eq 0) then 0 else $x * $y}) },
    $seq := (1 to 1000000) ! function($x as xs:integer) as xs:integer {$x - 15}(.)
 return
   $product($seq)

This, if laziness at all, is not intelligent, because as the saying goes:

"Efficiency is intelligent laziness" , which in this particular case it clearly isn't.

We achieve "intelligent laziness" when we define it to include both short-circuiteness and generators.

[dnovatchev]

Generators do seem to be a potentially very powerful tool, though probably only one for expert users. I can see this being made available as a function

fn:generate-sequence($len as xs:nonNegativeInteger, $get as function(xs:positiveInteger) as item()) where the result is a sequence of length $len, whose $Nth item can be determined by calling $get($N).

Sorry, but this isn't a generator at all.

A generator returns only one result at a time, and in never a complete collection needs to be in-memory.

What is proposed in the quote returns a complete sequence... which may never be wanted and is wasteful both in time and memory.

[michaelhkay]

significantly worse

275 seconds is worse than 260, but not significantly worse. The reason that we choose lazy evaluation by default is that if it's the wrong choice, the overhead is typically only around 10%, whereas if it's the right choice, the benefit is often an order of magnitude.

There are of course many facets to lazy evaluation, which is the reason that I think it has no place in the language specifications. The language specifications should talk only about the observable results of an expression. We could possibly talk about the performance complexity of operations, for example that $seq[17] must execute in constant time, but we must never talk about implementation internals.

The issue in this thread is not about lazy evaluation, it is about detection of fixpoint conditions in a fold operation.

[michaelhkay]

What is proposed in the quote returns a complete sequence

It returns a value that behaves exaclty like a sequence but one where each item in the sequence only needs to be materialised if and when it is needed (another form of lazy evaluation).

For example, you could use this to generate a sequence of total sales for day N since the opening of your business, where the $get function does a database access to find the required data, and the database access will only take place for a particular date if someone asks for the value for that date. If you want the total sales for a particular quarter, you could do sum(subsequence($sales, 15200, 90)) with the expectation that items in the sequence outside that range will never be evaluated.

[liamquin]

On generators, pseudo-random number generation is an example we already have in the spec.

On fixpoint detection... i wonder how common this is... i think of string and node manipulation as much more common than numeric work in XSLT and XQuery and XPath, and it's not really possible to do an early exit from string concatenation, for example. But maybe that's because i mostly work with documents.

I suppose one could throw an exception inside the function one passed to fold-right and catch the result value around the outside, although that feels rather heavy-weight.

[dnovatchev]

What is proposed in the quote returns a complete sequence

It returns a value that behaves exaclty like a sequence but one where each item in the sequence only needs to be materialised if and when it is needed (another form of lazy evaluation).

Sorry to quote your initial function definition again (the highlighting is mine):

fn:generate-sequence($len as xs:nonNegativeInteger, $get as function(xs:positiveInteger) as item()) where the result is a sequence of length $len, whose $Nth item can be determined by calling $get($N).

This function definition lacks a return type, and it seems that it cannot be described in a specification where such concepts as "materialized" and "lazy" are not defined.

And you initially said: "where the result is a sequence of length $len", but now we hear something rather different.

For a good starting definition of a generator, please see this comment (above)

[dnovatchev]

On fixpoint detection... i wonder how common this is... i think of string and node manipulation as much more common than numeric work in XSLT and XQuery and XPath, and it's not really possible to do an early exit from string concatenation, for example. But maybe that's because i mostly work with documents.

One example of such "fixpoint detection":

Whenever you call ChatGPT "stupid" or compare it unfavorably to Google's AI, it abruptly terminates the current conversation and starts a new one 😄

[dnovatchev]

The issue in this thread is not about lazy evaluation, it is about detection of fixpoint conditions in a fold operation.

We can define an evaluation strategy as "lazy" if it takes advantage of fixpoint detection and generators.

And this is exactly what the provided solution to the problem demonstrates.

Both of these are not just some abstract concept, but have been implemented in specific, executable XPath code.

I'm having a little trouble finding concrete use cases, however. One might be for sparse arrays.

The documented in code and implemented laziness, decreased the evaluation time of Saxon from 260 seconds to zero seconds -- an increase of infinite number of times...

And if such result is not a good use-case for you, then I doubt anything else would be.

[dnovatchev]

There are of course many facets to lazy evaluation, which is the reason that I think it has no place in the language specifications. The language specifications should talk only about the observable results of an expression. We could possibly talk about the performance complexity of operations, for example that $seq[17] must execute in constant time, but we must never talk about implementation internals.

Oh... Does God forbid us from using such dirty words as "laziness" and "short-circuiting" ?

One more proof that I am an atheist 😄

And so are the creators of Haskell (lazy), and C# (Deferred execution), and Python 3, Wikipedia ...

Will have the pleasure of meeting their creators ... in Hell 😄

Thanks to any zealous preacher reminding us of the ultimate predicament for any such blasphemy.

[michaelhkay]

Your Haskell link points to places where the term "lazy" is used, but I haven't been able to find where it is defined.

Your C# link points to documentation for users of the language indicating what they can expect when they use a particular construct; it doesn't describe conformance rules for implementations.

Your Python link points to a document that reads like a description of an implementation, rather than a language specification that implementations must conform to.

My concern here is that while we all have a general notion of what it means for evaluation to be lazy, many variations are possible in practice and it is difficult to be prescriptive about exactly what strategies come within the definition. If we use the term prescriptively then (a) we need a very precise statement of what we mean by it, and (b) we must avoid forcing implementations to adopt inefficient strategies. For example, I would hate to see anything that required processors to evaluate preceding::* from left-to-right rather than right-to-left.

[dnovatchev]

My concern here is that while we all have a general notion of what it means for evaluation to be lazy, many variations are possible in practice and it is difficult to be prescriptive about exactly what strategies come within the definition. If we use the term prescriptively then (a) we need a very precise statement of what we mean by it, and (b) we must avoid forcing implementations to adopt inefficient strategies. For example, I would hate to see anything that required processors to evaluate preceding::* from left-to-right rather than right-to-left.

I think that for now it would be good to define "laziness" as using shot-circuit-ness and generators.

[dnovatchev]

Here is the first example of a generator that produces an infinite collection, as requested by @ChristianGruen .

Here this means that the generator can be called unlimited number of times and on each call it returns the correct value of the requested member of the intended infinite collection.

I encourage everyone to copy the following code, then paste it into an XPath processor and execute it:

let $fold-right-sc-lifted-inner := 
        function ($seqGen as function(xs:integer) as array(*),
                  $index as xs:integer,
                  $zero as item()*,
                  $startLimit as xs:numeric,
                  $f as function(item(), item()*) as item()* ,
                  $fGetPartial as function(*),
                  $self as function(*)
                  ) as item()*
                  {
                    let $nextSeqResult := $seqGen($index),
                        $isEndOfSeq :=  $nextSeqResult(1),
                        $seqItem := $nextSeqResult(2)
                      return
                        if($isEndOfSeq) then $zero
                          else
                            if(function-arity($fGetPartial($seqItem, $zero)) eq 0)
                              then $fGetPartial($seqItem, $zero) ()
                              else $f($seqItem, $self($seqGen, $index+1, $zero, $startLimit, $f, $fGetPartial, $self))
                  },

    $fold-right-sc-lifted :=  
    function ($seqGen as function(xs:integer) as array(*),
              $startIndex,
              $zero as item()*,
              $startLimit as xs:numeric,
              $f as function(item(), item()*) as item()*,
              $fGetPartial as function(*) 
              ) as item()*
              {
               $fold-right-sc-lifted-inner($seqGen, $startIndex, $zero, $startLimit, $f, $fGetPartial, 
                                                                                           $fold-right-sc-lifted-inner)
              },

   $NaN := xs:double('NaN'),
   $INF := xs:double('INF'),

   $fSeq1ToN := function($ind as xs:integer, $indStart as xs:numeric, $indEnd as xs:numeric) as array(*)
                {
                  if($ind lt  $indStart or $ind gt $indEnd)
                    then  array{true(), $NaN}
                    else array{false(), $ind}
                },
   $fSeq-1-Infinity := $fSeq1ToN(?, 1, $INF),
   $fSeq-1-INF-minus-749 := function($n as xs:integer)
   {
     array{$fSeq-1-Infinity($n)(1), $fSeq-1-Infinity($n)(2) -749}
   },
   $fMult  := function($x, $y)  {$x * $y},
   $fMultGetPartial := function($x, $y)
   {
     if($x eq 0)
       then function() {0}
       else function($z) {$x * $z}
   }

   return
   (
     $fold-right-sc-lifted($fSeq-1-INF-minus-749,  1, 1, 1, $fMult, $fMultGetPartial)
   )     

As expected, evaluating this by an XPath processor correctly produces:

0

This is only possible due to the laziness of the function (to be renamed to $fold-lazy)

[ChristianGruen]

The early-exit approach turns out to be powerful. Here’s one more example with fn:fold-left to compute the first 10,000 prime numbers (takes 200 ms on my system):

fold-left(1 to 1000000000000000, (), fn($r, $i) {
  if(count($r) >= 10000) then $r else
    ($r, $i[. > 1 and (every $i in 2 to xs:integer(math:sqrt(.)) satisfies . mod $i)])
})

The iterate-while approach (with a positional argument):

iterate-while((),
  fn($r, $p) { count($r < 10000) },
  fn($r, $p) { $r, $p[. > 1 and (every $i in 2 to xs:integer(math:sqrt(.)) satisfies . mod $i)] }
)

And a classical recursive approach:

declare function local:generate($count, $current, $primes) {
  if (count($primes) >= $count) then $primes else (
    local:generate($count, $current + 1, (
      $primes, $current[. > 1 and (every $i in 2 to xs:integer(math:sqrt(.)) satisfies . mod $i)]
    ))
  )
};
local:generate(10000, 1, ())

[dnovatchev]

@ChristianGruen

iterate-while((),
  fn($r, $p) { count($r < 10000) },
  fn($r, $p) { $r, $p[. > 1 and (every $i in 2 to xs:integer(math:sqrt(.)) satisfies . mod $i)] }
)

I am getting this error: "_Function with 2 arguments supplied, 1 expected_."

[ChristianGruen]

I am getting this error: "_Function with 2 arguments supplied, 1 expected_."

The positional extension is not part of the standard yet (see https://github.com/qt4cg/qtspecs/issues/670#issuecomment-1703870411 / #516).

[michaelhkay]

declare namespace math = 'http://www.w3.org/2005/xpath-functions/math';
fold-left(1 to 1000000000000000, (), fn($r, $i) {
  if(count($r) >= 10000) then $r else
    ($r, $i[. > 1 and (every $i in 2 to xs:integer(math:sqrt(.)) satisfies . mod $i)])
})

What's the precise reasoning here? I guess something like:

• The function body is of the form if (C) then $r else E
• C does not depend on $i
• Therefore, if C is true on one iteration then it will also be true on all subsequent iterations

How does this translate to fold-right? Dimitre's original example used the callback function

if ($x eq 0) then $x else $x * $y

which is testing whether an individual item in the sequence is zero, not whether the accumulated result is zero. Knowing that you can exit when an individual item is zero seems to require knowledge of the semantics of multiplication; it would be wrong to do an early exit if the expression E were $x + $y.

[ChristianGruen]

What's the precise reasoning here?

It's similar to your summary; see https://github.com/qt4cg/qtspecs/issues/670#issuecomment-1703916888.

How does this translate to fold-right?

The optimization can be applied to both left and right folds. For right folds, the check needs to be performed after the modifying operation. And you’re completely right, the check depends on the computed value, not the iterated one.

[ndw]

@dnovatchev the example in what GitHub identifies as issuecomment-1704424043 above is large and complicated and apparently produces the result 0. What on earth does it do? What it is attempting to implement appears utterly opaque (to me) from the code.

[dnovatchev]

declare namespace math = 'http://www.w3.org/2005/xpath-functions/math';
fold-left(1 to 1000000000000000, (), fn($r, $i) {
  if(count($r) >= 10000) then $r else
    ($r, $i[. > 1 and (every $i in 2 to xs:integer(math:sqrt(.)) satisfies . mod $i)])
})

What's the precise reasoning here? I guess something like:

• The function body is of the form if (C) then $r else E
• C does not depend on $i
• Therefore, if C is true on one iteration then it will also be true on all subsequent iterations

@michaelhkay I studied @ChristianGruen 's approach. It seems that he performs a break from fold-left after all the useful work is finished. For example: one doesn't know how long list of natural numbers to have, such that it will certainly contain the first N primes. This is why Christian specifies a list of some tremendous length (although this is still just a guess!) and makes the passed function stop doing any new work after the Nth prime is found -- which is what his code detects and then performs the break.

In contrast, the short-circuiting in the proposed fix to fold-right happens before any work is done at all - because the computations only start when the last member of the collection is reached, but the value causing the short-circuiting is found before that. Thus, no guessing, no need to specify unnecessary length - as we can simply pass a generator that generates an infinite collection, and thus we don't have to guess how long a collection to process.

How does this translate to fold-right? Dimitre's original example used the callback function

if ($x eq 0) then $x else $x * $y

which is testing whether an individual item in the sequence is zero, not whether the accumulated result is zero. Knowing that you can exit when an individual item is zero seems to require knowledge of the semantics of multiplication; it would be wrong to do an early exit if the expression E were $x + $y.

[dnovatchev]

@dnovatchev the example in what GitHub identifies as issuecomment-1704424043 above is large and complicated and apparently produces the result 0. What on earth does it do? What it is attempting to implement appears utterly opaque (to me) from the code.

@ndw This is exactly the same example as in the final section of the opening comment, but instead of passing a generator that generates the sequence $fSeq-1-1M-minus-3, which only generates one million items from 1 to 1000000, with 3 subtracted from each:

-2, -1, 0, 1, 2, 3, 4, 5, 6, ..., 999995, 999996, 999997

here we create a generator whose upper limit is $INF , that is xs:double("INF") .This generate will never stop to produce new results, because for any integer $ind it is always true that $ind lt $INF

But because we know that there is a 0 generated (this time 749 is subtracted from any 1 to $INF, so the sequence starts from -748) :

-748, - 747, -746, ... -3, -2, -1, 0, 1, 2, 3, ... , 99999999999999999999999999999999999, ... , ...

I would be happy to answer any other questions that you may have!

[ChristianGruen]

@michaelhkay I studied @ChristianGruen 's approach. It seems that he performs a break from fold-left after all the useful work is finished.

Exactly. The loop 1 to 1000000000000000 is obviously a hack. It would be a bit cleaner to iterate until $max * $max, but the other two solutions are the ones that I certainly prefer. A better use case could be to iterate over a node sequence (but it’s tricky to give a compact example).

In contrast, the short-circuiting in the proposed fix to fold-right happens before any work is done at all

I believe you indicated that the short-circuiting approach would also work for computing prime numbers, right? How would this look like?

[ChristianGruen]

But because we know that there is a 0 generated (this time 749 is subtracted from any 1 to $INF, so the sequence starts from -748) :

It's currently easy to trigger a stack overflow error if a larger value than 749 is chosen. Do you think this can be fixed, or is this approach limited to small numbers?

I agree with Norm that an intuitive use case would be of great help to motivate the usefulness of the approach.

[dnovatchev]

I believe you indicated that the short-circuiting approach would also work for computing prime numbers, right? How would this look like?

@Christian Yes, I must provide this soon, but I just woke up at 9:43AM (overslept again), so everything in its due time 😄

I actually plan to show how to implement a function like:

take n xs

that produces the first n members from the list xs , which can be an infinite list.

So, we will have two separate generators of infinite collections :

1. Generator 1: 1 to $INF - that is: all natural numbers

2. Generator 2: Uses Generator 1, checks if the current result from Generator 1 is a prime and if so, produces this prime as its own current result. Thus Generator 2 produces the infinite collection of all prime numbers

3. A client (intended to use fold-lazy) that uses Generator 2 and from this infinite collection of primes only takes the first N.

[dnovatchev]

It's currently easy to trigger a stack overflow error if a larger value than 749 is chosen. Do you think this can be fixed, or is this approach limited to small numbers?

It depends on the amount of memory one has and on the configuration of the Processor. We need to figure out how they avoid this in Haskell.

Another approach is to use the DVC (Divide and Conquer) strategy, which in this case would be to process the input in smaller chunks and to replace each chunk with the result produced on it.

I agree with Norm that an intuitive use case would be of great help to motivate the usefulness of the approach.

I already described a general use-case where we need to find the first N members of a collection that have some common property (like being primes) and where we don't have (and don't want) to guess how large the input collection needs to be in order to contain all such N members (and not to overspecify it's possible size, which again is just having to guess and going possibly wrong with such a guess),

[dnovatchev]

It's currently easy to trigger a stack overflow error if a larger value than 749 is chosen. Do you think this can be fixed, or is this approach limited to small numbers?

It depends on the amount of memory one has and on the configuration of the Processor. We need to figure out how they avoid this in Haskell.

Actually, the stack overflow only happens in BaseX, and I just checked with really longer sequence that this still works in Saxon.

This works great in Saxon (the position of the short-circuiting value is 4000):

And only raises a stack-overflow exception in BaseX - even when the short-circuiting value is just at position 750 ):

Conclusion: Even at present XPath processors do not produce stack-overflow when traversing with fold-lazy a significant part of the provided input collection.

Most probably they could be configured to allocate more stack and heap space (whatever is being consumed by their JVM) which would result in sufficient greater resilience to the length of the input that needs to be traversed.

So, it is not the amount of RAM on ones machine (I have 32GB that should be sufficient for most practical cases), but the proper configuration of the XPath processor.

[benibela]

It depends on the amount of memory one has and on the configuration of the Processor. We need to figure out how they avoid this in Haskell.

Haskell is infamous for having space leaks