Provide an iteration protocol

[PuercoPop]

Currently the iterables are a 'closed abstraction'. An iterator protocol would allow the user to 'teach' the for tag how to iterate different the objects.

I propose something similar to python iterators. a next generic function and a stop-iteration condition. An addition to consider would be that the iterator knows how many values per iteration it provides. Although just using values could be a better choice.

Ideally this would help to break up the implementation of :parsed-for which is currently too large.

Is this extension desirable?

[mmontone]

I suppose that would be nice, yes. I would implement a separate CL iterators library first, and then bind Djula to it.

[mmontone]

@PuercoPop What about this iterators implementation?: https://github.com/bytecurry/colliflower

Maybe it's worth a look.

[PuercoPop]

It looks promising!

[PuercoPop]

I recently became aware that cl-enumeration provides an iteration protocol as well. But after checking out [grid-generators] I think the generator idiom* they use is sufficient for our use case.

We could provide a generic function make-genetator with (iterable &optional (eoi))` as arguments. It would straight forward to provide an implementation for the corresponding classes of the types in iterable-list but I'm a little stumped on how to transform the parsed-for token-compiler to use the idiom.

Do you like the idea?

The generator Idiom*

Just to clarify the generator idiom used by grid-generators is to bind to a generator in a with clause and then call the generator successively in a :for/:= clause. Using eoi to signal the end of iteration. This last value is necessary due to the plausibility that nil would be a value of the collection.

[jorams]

For such generators, I think a better solution for allowing nil as a value of the collection would be to use multiple values. Something like (values element present-p), where present-p is nil at the end of iteration, in the same way gethash tells you whether the value you're getting back is an actual value from the hash table or just a default.

[PuercoPop]

@jorams Although the multiple values would simplify the test to end the iteration, the problem is that, to the best of my knowledge, the :for/:= does not support multiple-value-bind, only destructuring. That is why I proposed a solution similar to read/peek-char/etc instead of gethash.

[mmontone]

Yes. I think it is a good idea to implement loops with some kind of generic iterator. So, what's the idea, use cl-enumerations, or grid-generators? Why not https://github.com/bytecurry/colliflower

Is there any other library we can consider?

[PuercoPop]

I was proposing to roll our own in the spirit of grid-generators

[mmontone]

Ah. Fair enough. Maybe an strategy could be that you implement the iterators protocol and implementation, and I can try to itegrate them later to Djula. What do you think?

[mmontone]

Although, I think colliflower iterators look quite complete and usable, and could try to integrate that straight away...

[mmontone]

Just playing:

CL-USER> (liter:get-iterator (make-hash-table))
#<CLOSURE (LAMBDA () :IN LITER/BASE:GET-ITERATOR) {1007ACBD7B}>
To load "anaphora":
  Load 1 ASDF system:
    anaphora
; Loading "anaphora"

CL-USER> (let ((table (make-hash-table)))
           (setf (gethash :x table) "foo")
           (setf (gethash :y table) "bar")
           (let ((iterator (liter:get-iterator table)))
             (loop :for (value found-p) := (multiple-value-list (liter:inext iterator))
                :while found-p
                :do (print value))))

(:X . "foo") 
(:Y . "bar") 
NIL

Note the use of multiple-value-list

[PuercoPop]

Ah. Fair enough. Maybe an strategy could be that you implement the iterators protocol and implementation, and I can try to itegrate them later to Djula. What do you think?

That is what I was thinking, I'll work it on the weekend.