Guidance on future of this library and research

[solvingj]

I am also very interested in using/supporting/promoting this project. Despite it being "under active development", I went ahead and packaged it for Conan.io package manager.

https://github.com/bincrafters/conan-libhandler

@daanx Would you be willing to comment on the current trajectory of the library, and your goals for it as an OSS project?

It would be very helpful to developers who might want to try using this library, contributing to it, and/or promoting it. I think that if more people heard about this library, it would get a LOT of attention very quickly, but perhaps it's not the right time for that yet. It's hard to know, because it's very unusual for such a library to appear and be "effectively complete" in one month, and there has been no further commits or discussion for 5 months. In June, the technical report said that the work was planned to continue via an internal implementation, and the README disclaims that it's under active development and subject to change.

An updated status would be highly appreciated, but in any case, thanks for such an interesting contribution!

[daanx]

Hi @solvingj ,

Good to hear you like the project. I am currently in Suzhou China giving a talk about the library at APLAS 2017. libhandler is a somewhat special library as it is more of a runtime system extension, like a garbage collector. As such, I don't think the internals will change much -- it either works and implements algebraic effects, or it doesn't; there is not much to add or change to its essence.

At this point, I consider the core of the library as "finished" (libhandler.c) in the sense that it works and implements every aspect needed to model algebraic effects and handlers. The main remaining work items are:

1. A nice standard library with common abstractions: in particular exceptions and exception handling, ambient state (implicit parameters), and a good async/await core library.

2. Have the exception handling defined in 1 work well with C++ too and write a nice C++ standard library too; I think classes can help quite a bit to make the definition of custom handlers more easy (where operations are defined by virtual methods).

3. A nice wrapper library for the API's of LibUV : this should reuse the exceptions of 1. and also build on the paper on async effects I wrote to implement robust and composable cancelation and timeout. There is a giant opportunity here for someone to basically create "NodeC" where it becomes convenient to write high performance webservers in C in a way that is very convenient.

4. Trying it out on more platforms and ensuring the makefiles etc work well everywhere.

5. Integrating this with the Boehm Weiser garbage collector to ensure this library can be used for runtime systems for languages like Scheme etc.

One thing I want to keep working on is to keep the library and essence small with few dependencies -- it should be very easy to integrate this into existing projects.

Now, being a researcher I cannot do all those things by myself easily :-) But I think it is probably best if I make an initial attempt at 1 and 2 since these are still so closely related to the overall design -- from that point on the library would be much more friendly to use out of the box, and it would give enough guidance for people to build on and extend (especially with regard to resource management). I hope to find someone who will attempt to do 3 where I can help with advice. Currently, the dev branch contains a nice libuv example but you can see that that really needs a more standard approach to exception handling which is still lacking.

Anyway, enough rambling, I need to attend talks again -- but I hope this helps and lets make libhandler great!

Daan

[solvingj]

Daan,

Thank you for rambling. I am very encouraged by this response. I have many followup questions and comments. I will list them here, although perhaps there's a better channel for future communication.

1. To what degree can/should this library (and future extensions) mimic Koka? I can't speak intelligently about Koka, but it's obviously the basis for this library, and it seems focused on practical programming problems. It's also nicely documented and reasonably mature. If it is indeed a good model for libhandler, it could give potential contributors who wish to extend the library something to code to. For example, how a Maybe or Optional type should be implemented, how verbs should be named, etc.

2. I understand the idea of using classes in this case. I spent quite a bit of time using functional paradigms in Java and C# where classes were the common coin. To me, the use of classes in this way always seemed forced, and led to awkward philosophical questions and situations, and sometimes real problems. In C++, some of the constructor quandries even seem much worse. I had hoped that the use of classes should be avoidable outside of class-based languages, but I don't have any concrete examples of alternatives to your goal. I will do some research here.

3. Absolutely, a practical and coherent combination of this library and libuv to create a functional async await framework in C seems to have unspeakable potential. Indeed, a "NodeC" would probably grab the most attention for such an implementation, but I have several others in mind too. I would think it wise to reach out to both libuv and libevent teams, to see if either would be interested in collaboration. I'm not an expert in the differences, but they seem reasonably similar in purpose.

4. You mention the build, and a desire to make it easy to integrate this library into existing projects. In the past 6 months, I've been working with a team of C/C++ devs building and packaging about 200 C and C++ libraries in order to make them easy to consume and integrate with. This has given us some unique perspective on the build tools and integrations. In short, regardless of our personal histories and opinions about CMake, it has proven to be an invaluable building block for downstream integration and consumption by modern projects and tools. I hope you will consider (or reconsider) implementing it as the primary build tool for this and related projects. If not, I understand, as the "CMake" question comes with several real technical (and sometimes political) implications.

5. The possibility of garbage collection seems an independent, but equally significant dimension of it's own. I think it would be best if it could be made optional somehow, which might be inconceivable. However, because of the unique characteristics of the functional paradigm used herein (compose-able, extension-methods), maybe it's possible to have a GC version which wraps the core version (which ideally would have no concept of GC). I'm not sure any of that is realistic though.

Thanks for taking the time to respond so thoughtfully. I will now be spending significant time reading the code and discussing with others.

[solvingj]

I have taken the example from the research paper and created a separate Github project for it.

https://github.com/solvingj/libhandler-example

I am looking for guidance as to what to put in the License file. While the libhandler library is licensed under Apache 2.0, the code in this project was copied from the research paper itself. For that, I was unable to find a license or copyrighted. I seem to be buried in OSS licensing conundrums lately.

[solvingj]

@daanx In your fugure goals, you did not mention a Maybe or Option type. What are your thoughts on implementing one of these types alongside the Exception type?

[mattgreen]

@solvingj TBH I think Maybe/Option is not related to this library, as they track the presence of values via types, which is more a language concern, whereas libhandler is concerned with handling of side effects for an implementation of a language. libhandler is effectively a sort of substrate for languages.

[solvingj]

I don't disagree in principle. I personally just enjoy working with these types and the patterns that arise from them, and would like to use them in conjunction with libhandler. Also, they share a need for parametric polymorphism with this library, and so I think it would be novel to see them implemented with the same "strategy" used here, and then see what some real world code looks like when using them all together. I'm getting closer to a time when i can do such experimentation. Hopefully this year.

[solvingj]

In any case, based on github activity for this project and @daanx , this library and related ideas around it don't seem to be a current focus. It seems that the initial academic/research goals were reached and the effort and interest may now be mostly concluded. I remain interested in working with it personally, but am closing ticket as a result.

[daanx]

Hi all, good to see the interest in better abstractions :-) I have been busy lately thinking about correct exception handling and finalization of resources -- very important in the context of C :-) I wrote a (technical) paper about this:

https://www.microsoft.com/en-us/research/publication/algebraic-effect-handlers-resources-deep-finalization

and I want to update the library to reflect the mechanisms of that paper. In particular, this can lead to nice abstractions to manage resources, like memory or files, in nice ways; much like RAII but then directly in C.

Besides doing that, the big thing on the agenda is to extend the library to provide a set of common abstractions for Exceptions (and resource finalization), Scoped state, Implicit parameters (i.e. scoped read-only state), for example, the current web request object, Async/Await, and perhaps iteration. That will be good starting point for others to start using this library and extending it with further abstractions.

(Actually, I am currently trying to find resources to help me do it.. I will know more about this in about 2 weeks. )

With regard to Option,Maybe: there is always an interesting tension between returning results as maybe types, or as a regular type and throwing exceptions. I think it usually depends on the application what is the best choice: if you find yourself writing lots of boilerplate to continuously check result types and propagate errors, an exception effect is better I think since you are basically manually writing the binds of the maybe monad otherwise. In essence both approaches are isomorphic since we can always handle a function with exceptions (of type a -> exc b ) to become an explicit maybe returning function (of type a -> total maybe<b>), .. and the other way around. My plan is to write an explicit Exception effect abstraction but also provide the to_maybe handler.

[daanx]

Oops, ha, just closed it :-)

[daanx]

I'll reopen it for little while longer to ensure my reply didn't get lost..

[solvingj]

Good luck finding help, I hope you do. I will read the paper today despite my lack of comprehension of the math. The mention of TCO was exciting to see, but the finalization problem is much more relevant to me. Since I discovered the library, I have been regularly thinking about whether or not the handler abstraction (and perhaps some additional/related ones) can ever lead to new and deterministic patterns for wrapping around malloc/free/realloc in a safe and generic way.

WRT Maybe, I agree that guidance on effective use of them should depend substantially on the problem domain. After seeing your implementation of a Exception and Maybe, the next goal for me would be to create Result and Exceptional based on them. Also, assuming you include bind for Maybe, I would follow that.

There's a major outstanding question which is whether or not the technique you use for polymorphism in this library is broadly useful, and whether or not you will use it in implementing a generic container like Maybe. I don't know know enough about the performance implications and optimizations that might happen. Can you comment on any of this?

[solvingj]

We can close this if you like. I'll continue to watch to see if the skip frames and other optimizations in the most recent paper get implemented.