How to Handle End of Stream

[ssadedin]

I have a few common use cases that I'm finding hard to deal with. The most obvious one is I want to process lines of a file. It's all fine except that I don't know how to ensure the file is closed once the stream is finished. If I pass Stream objects around I have no way to ensure that. Intuitively, what I want is a syntax like this:

def foo() {
    def r = new File("test.txt").newReader()
    return new Stream({ f.readLine() })
        .filter { blah }
        .map { bloop }
        .end { r.close() }
}

Then a client of this function can call:

foo().map { fizzle }.until { blag }

And the key point is, the "end { }" closure is guaranteed to be called when the stream terminates, regardless of whether it's because the file was exhausted or whether it's because someone downstream imposed an "until".

Is there a way to handle this right now, or is there a possibility to add such an "end" function?

[timyates]

Hiya!

Don't think there's a way to do this currently :-(

If you can do all the processing inside foo then obviously you could just use withReader, but that doesn't help with this use case

I'll have a think, I can't think of a similar method in other iterator/stream frameworks and wonder if there's a reason.

Tim On 25 Jun 2015 03:01, "Simon Sadedin" notifications@github.com wrote:

I have a few common use cases that I'm finding hard to deal with. The most obvious one is I want to process lines of a file. It's all fine except that I don't know how to ensure the file is closed once the stream is finished. If I pass Stream objects around I have no way to ensure that. Intuitively, what I want is a syntax like this:

def foo() { def r = new File("test.txt").newReader() return new Stream({ f.readLine() }) .filter { blah } .map { bloop } .end { r.close() } }

Then a client of this function can call:

foo().map { fizzle }.until { blag }

And the key point is, the "end { }" closure is guaranteed to be called when the stream terminates, regardless of whether it's because the file was exhausted or whether it's because someone downstream imposed an "until".

Is there a way to handle this right now, or is there a possibility to add such an "end" function?

— Reply to this email directly or view it on GitHub https://github.com/timyates/groovy-stream/issues/21.

[ssadedin]

Thanks for the followup - yes, I'm not sure what the right solution is. TotallyLazy seems to have some functionality along these lines, but I can't see any examples of how it works - see:

https://github.com/bodar/totallylazy/blob/master/src/com/googlecode/totallylazy/Closeables.java

JDK8 streams seem to sidestep the issue by putting a "close()" method onto the Stream interface itself:

https://docs.oracle.com/javase/8/docs/api/java/util/stream/BaseStream.html

But that just means the consumer of the stream has to close it. If nobody else is doing it then It might be that what I want is an anti-pattern. But it seems intuitive to me that being able to functionally compose streams is a desirable thing.

[timyates]

I didn't realise this was a thing with Java 8 streams :-)

I'll take a look into adding onClose and close into Groovy streams... As they are just a chain of iterators, it should be possible to walk up the parent, and silently try to close any resources that have been captured...

Will need to make sure we don't kill auto-closeable AND grooy's withReader/InputStream though ;-)

Tim

On 25 June 2015 at 13:38, Simon Sadedin notifications@github.com wrote:

Thanks for the followup - yes, I'm not sure what the right solution is. TotallyLazy seems to have some functionality along these lines, but I can't see any examples of how it works - see:

https://github.com/bodar/totallylazy/blob/master/src/com/googlecode/totallylazy/Closeables.java

JDK8 streams seem to sidestep the issue by putting a "close()" method onto the Stream interface itself:

https://docs.oracle.com/javase/8/docs/api/java/util/stream/BaseStream.html

But that just means the consumer of the stream has to close it. If nobody else is doing it then It might be that what I want is an anti-pattern. But it seems intuitive to me that being able to functionally compose streams is a desirable thing.

— Reply to this email directly or view it on GitHub https://github.com/timyates/groovy-stream/issues/21#issuecomment-115236585 .

[timyates]

So, I had some free time tonight, and took a stab at adding a close() method to streams to try and close all sources being used by the stream...

It ended up being a bit of a beast

Got all existing tests passing, and added a solitary test for the new close method

But as the commit message says, this commit feels like it needs a lot of looking at until it can be trusted...

I guess you were more looking for an onFinished method though, to attach a closure/Function to be run when the stream has concluded?

which I guess is another commit, on another free night ;-)

[ssadedin]

This looks like a great start, and certainly helps to manage the issue. I have cloned / built the feature branch and will try and exercise it over the next few days. It worked for my simple tests already.

One minor note - this made it depend on Java 1.7 for me. Not a big deal, I think, but until now I could run it on JDK1.6, so thought I'd mention it in case that's important.

A generalised "onFinished" type feature would definitely still be a great thing to have!

Thanks!

On Mon, Jun 29, 2015 at 6:49 AM, Tim Yates notifications@github.com wrote:

So, I had some free time tonight, and took a stab at adding a close() method to streams to try and close all sources being used by the stream...

It ended up being a bit of a beast https://github.com/timyates/groovy-stream/commit/7242e70af564f32372b87062d9f786bb7c27e574

Got all existing tests passing, and added a solitary test for the new close method https://github.com/timyates/groovy-stream/blob/7242e70af564f32372b87062d9f786bb7c27e574/src/test/groovy/groovy/stream/ReaderTests.groovy#L75

But as the commit message says, this commit feels like it needs a lot of looking at until it can be trusted...

I guess you were more looking for an onFinished method though, to attach a closure/Function to be run when the stream has concluded?

which I guess is another commit, on another free night ;-)

— Reply to this email directly or view it on GitHub https://github.com/timyates/groovy-stream/issues/21#issuecomment-116333211 .

[timyates]

Cool, thanks for trying it all out :-)

Yeah, adding AutoCloseable (so it can be used in a trywithresources query), has pushed it to Java 7... My main thought is that as Java 7 is already EOL, that's fine, but I guess others may disagree, and it may reduce the possible userbase

I'll have a look at onFinished :-D

On 30 June 2015 at 14:07, Simon Sadedin notifications@github.com wrote:

This looks like a great start, and certainly helps to manage the issue. I have cloned / built the feature branch and will try and exercise it over the next few days. It worked for my simple tests already.

One minor note - this made it depend on Java 1.7 for me. Not a big deal, I think, but until now I could run it on JDK1.6, so thought I'd mention it in case that's important.

A generalised "onFinished" type feature would definitely still be a great thing to have!

Thanks!

On Mon, Jun 29, 2015 at 6:49 AM, Tim Yates notifications@github.com wrote:

So, I had some free time tonight, and took a stab at adding a close() method to streams to try and close all sources being used by the stream...

It ended up being a bit of a beast < https://github.com/timyates/groovy-stream/commit/7242e70af564f32372b87062d9f786bb7c27e574

Got all existing tests passing, and added a solitary test for the new close method < https://github.com/timyates/groovy-stream/blob/7242e70af564f32372b87062d9f786bb7c27e574/src/test/groovy/groovy/stream/ReaderTests.groovy#L75

But as the commit message says, this commit feels like it needs a lot of looking at until it can be trusted...

I guess you were more looking for an onFinished method though, to attach a closure/Function to be run when the stream has concluded?

which I guess is another commit, on another free night ;-)

— Reply to this email directly or view it on GitHub < https://github.com/timyates/groovy-stream/issues/21#issuecomment-116333211

.

— Reply to this email directly or view it on GitHub https://github.com/timyates/groovy-stream/issues/21#issuecomment-117172869 .