The road ahead (where to go?)

[halgari]

For some time I've been working on the effect branch. And although I've made some progress, it's diverged pretty far from Pixie and from Clojure. So far infact that I don't see it ever being close enough to Clojure to port any code.

So here's the question: Do we want "native Clojure", that's super fast, and has all the pros/cons of Clojure (no tailcalls, no deep yield, no stacklets, no lightweight threads, etc.) or do we want all of this, but also something that's not even close to Clojure.

I'm seriously considering dropping the effect changes and going with a much more standard lisp, perhaps even embracing the Clojure roots more, focusing on compatibility. Have thoughts? Voice them here.

[heyLu]

Can you expand a bit on what the tradeoffs are? E.g. what wouldn't be possible if we adopt the "effect way"? I must admit that the things in the effects README sound quite appealing, but I'm not familar with the theory behind it, so if I should read a bit more first, please say so.

[halgari]

Well the effects branch has the possibility to become much more pure. No mutability anywhere, anytime. All mutability becomes effects that must be handled outside the main program loop. It's transparent to the user though. So think something like the state monad. You're adding data to some opaque thing that exists outside your control.

But this means we can't support atoms, refs, etc. These things instead become IDs. Then when you call swap, the id is passed up the stack to the handler. And the work is done there. The same applies to transients and deftype...well actually deftype would never be mutable.

This means however that we can support lots of things that require pausing the interpreter. Stacklets, deep generators, etc. All "just work".

Getting this all to work with RPython has been and will continue to be a bit tricky. Currently the JIT freaks out and refuses to compile certain traces, I have an idea of how to maybe fix that, but I'm starting to wonder if I should take a step back and re-think these goals again. A interpreter like this can be rather fast (probably not faster than pixie is now), but it's a new way of programming, and thinking of programming. But more importantly, porting would become mostly impossible, and we'd have to port all of stdlib. And un-jitted code becomes much much slower.

So the other option is to just say "hey, let's be less flashy and and instead focus on making something that's more like Clojure from the start".

From a lot of the interest I'm seeing, people seem to care more about Pixie being fast and native, than anything else. So I'm thinking about these things.

[andrewchambers]

I still think pixie could have tailcalls and lightweight threads if it does not use the cstack as its own stack. With or without that, personally I think the effects branch is way too "out there" in terms of what people are willing to adopt.

[halgari]

Sadly it's not as simple as not using the cstack. Take for example, PersistentHashMap, this requires eq and hash, but it requires them inside HashCollisionNode. At that point, if hash decides to pause the current thread, you have to save the pixie stack and the RPython stack for the code inside HashCollisionNode. That's why stacklets exist. The problem with stacklets is that they are basically copies of the cstack that the GC understands how to read. Mucking with the cstack tends to cause segfaults...

[andrewchambers]

Hmm, I must admit, its a bit confusing to me, I don't know how scheme implements some of these things generally. (Is there an rpython scheme which supports tail calls?) Regardless, clojure semantics are still quite powerful without them.

I don't fully understand the effects branch, which is probably the same for most people.

[halgari]

Another option I just thought of would be to extend the existing yield bytecode to allow for multiple pixie functions to be paused. We would then throw an exception if such a yield/async take/async put/ etc., showed up inside a NativeFn. I think the cases this would happen would be pretty rare, so something like this would be disallowed: (deftype Foo [] IObject (-eq [a b] (<! c))) but you could still do something like this: (deftype Foo [] IObject (-eq [a b] (= b (generator (yield 1) (yield 2))))) The first one requiring native code to be paused if it attempted to call -eq on this object. While the latter is just a function that uses these features internally. Hrm...I'll have to give this a try on a new branch of master and see. Shouldn't take too long to hack out.

[gigasquid]

I think there is a great desire to have a fast native Clojure like lang for scripting.

That said, the work you have done with the effects branch sounds great.

Every new lang comes with the "library" problem. It sounds like one path of solving that is making libraries easy to port from Clojure. Perhaps there are other solutions?

[arrdem]

So this is just my 0.02, and I don't have any code in pixie so take it for what it's worth.

I am personally interested in a pure, effects system based language derived from Clojure even if it looks nothing like the parent language. This is the direction that my own explorations for Oxcart, Oxlang and Kiss are all kinda heading. So in either case I'd be curious to see $NOT_CLOJURE_LANG even if it turns out not to be the future of Pixie.

As to Pixie, I would say that there is probably more short term value in having a (near) Clojure target for Python interop and fast native scripting as @gigasquid points out.

[devn]

First off, thanks for even asking!

This is your language, and I think you should do with it whatever you feel is best. But if you're asking me what I'd want, closer to Clojure is the way to go. From an adoption standpoint it makes a ton of sense. As @gigasquid and @arrdem mentioned, there are a lot of people who would love to have something like this in their back pocket, myself included.

Either way, keep up the great work. Pixie just finished building. Looking forward to digging in.

[brandonbloom]

do we want all of this, but also something that's not even close to Clojure.

It would be great to have an alternative Clojure implementation with fast C FFI, but I'm not particularly convinced that it would be meaningfully different than using JNA for server use cases and I'm not convinced that Clojure is a particularly appropriate language for scripting use cases. At best, Pixie becomes a Clojure implementation with the advantage that it would be quicker startup time.

That said, C FFI and quicker startup could both reasonably be provided by investing in ClojureScript on Node.js.

So my vote is to go for the grand experiment, but I'm mostly just an observer and (selfishly) I want to learn from others attempts at effects et al.

But this means we can't support atoms, refs, etc. These things instead become IDs. Then when you call swap, the id is passed up the stack to the handler. And the work is done there.

Why does this mean you can't support them? It just means you can't support direct field access to them, since they are concretely represented as IDs. You can get the exact same API as clojure by defining construction in terms of a unique-id generating effect plus the reset/swap/alter/etc effects specific to each identity.

The same applies to transients and deftype...well actually deftype would never be mutable.

Doesn't deftype produce getter methods? Couldn't you allocate fresh IDs on construction and define the getter methods in terms of effects?

One other thought: My (original) intention for EClj was to start by being a Clojure in the same way that Racket started as being a Scheme. Modern Racket is barely recognizable as a standard Scheme, but it still runs standard Scheme with a startup flag, since Racket hasn't really diverged an unacceptable amount. I do realize, however, that this means the implementation effort increases by a factor of 1 < N <= 2 + ε

[heyLu]

Overall, I'd like Pixie to be closer to Clojure, but full compatibility isn't important to me. As I said before, I think it would be very useful to be able to port libraries such as core.match, core.logic, test.check and probably even core.typed without having to rewrite them from scratch.

I think there is a place for a modern native lisp with persistent datastructures, not just because of faster startup than Clojure, but also by having a good dynamically typed language and a better development workflow. Bodil Stokke mentions that near the end in her recent The Future of Clojure talk. I think ClojureScript could only provide this if it would become self-hosting, which I don't see happening.

All that being said, I'm still curious if there isn't a way of having both the effects work and still being relatively close to Clojure. (Where "relatively close" means "in spirit", not full compatibility.)

[bmillare]

Given how difficult it has been for me to try to understand the effects branch with the current resources available including the paper, PyPy toolchain, and the documentation listed here, I am concerned that development of the effects branch code will mainly be a single man operation. I assume that a robust project will require at least more than one set of eyes that deeply understand the implementation and its implications on performance and improvement on the programming paradigm.

I feel that while the start of a discussion is necessary now, I don't think the decision about what path to take can be confidently made now because not enough users understand the effects branch and its implications. How can I decide if I want the effects branch if I never wrote code that used these features before?

This is why I think more time should be dedicated to the effects branch, to see its abilities in full form, and then communicate these benefits (and downsides) with several examples, so that we may come to an understanding. I think its too early to be swayed by compatibility. In the true Clojure spirit, we should strive to leverage the advantages of the platform, and in this case, that means taking full advantage of the PyPy toolchain.

My takeaway would be this, a new goal should be to create more articles that flesh out the usage of the effects branch features. While potentially a lot of these resources that describe the benefits are available somewhere, its still difficult to narrow what is important for this implementation. I believe more articles with basic examples and their references that describe the theory behind it, would go a long way in helping the pixie community understand.

[halgari]

Okay, so I'm calling this now. I think it's best to reign in the more interesting aspects and focus on making Pixie fast, and usable.

I finished up a patch today that enables threads and removes stacklets from Pixie (as they were rather unstable). From there I'll be focusing on FFI support and better IO (libuv or not is yet to be determined).

[arrdem]

:shipit:

[andrewchambers]

@halgari Nice - After FFI and IO is hammered out, A focus making the core std api useful and well designed is extremely important imo. I feel like some functions in there at the moment are hapharzardly added, rather than well designed.

Go is a great example of a well designed standard library which was then able to make backward compatibility guarantees. We don't want to have a bunch of crap functions in the std library that are constantly getting deprecated after we realize they don't provide the right levels of abstraction or features. Of course, we need an idiomatic pixie library and not a Go one, its just an example.

We need an idiomatic pixie library for at least the following (in no particular order):

• ffi
• basic algorithms
• regex.
• networking. UDP,TCP - IPV4/IP6
• directory traversal.
• file io. (we can start with slurp and spit, but fine grained file handles is needed too)
• subprocesses. (need access to streaming stdin, stdout, stderr as well as python style check_output) With bonus features:
• smtp
• ssh
• http server (threaded? event based?)
• event loop?
• checksums
• archives and compression
• encryption.

I'd like to stress naming and keeping the documentation for each on par with the quality of python std library documentation. (Easy to get an overview, easy to understand whats present).

Let me know anything that is missing.

[gigasquid]

:sparkles: