Provide native multi-core concurrency and support scalable hardware strucutures

[HowardMei]

Hi Jukkal,

Have you heard about software defined network (SDN) and multiple ARM core cluster server?

From the roadmap of mypy, it seems you're more interested in another general purpose implementation for python like pypy, which is interesting but might constrain the practical use of this project due to the burden of supporting vast legacy python code. Pypy team takes years' of effort to scratch the surface of the problem.

Do you have any intention of creating a python compatible/flavored DSL language to combine the elegance of python, the multi-core concurrency of Erlang and the type safety and many goodness of Haskell? It could be a killer language in my opinion :)

Why do we need above features in mypy more than others, like JVM/iOS/Android support?

• Java stack is too heavy and resource hungry to ARM based embeded systems.
• PyPy solves the performance issue for general computing but the attempt of taking GIL away is not promising. Concurrency provided by STM costs too much. Pypy uses a lot of memory too.
• Erlang does a nice job in concurrent computing and embeded system but it's hard to learn and the single core performance is not good enough. Though there are some interesting dialects to make it fancier: http://www.efenelang.org/ or http://reia-lang.org/
• Haskell abounds with features but it's not for soft real time systems like SDN Controllers/Embeded devices which will be the keys to future laterally scalable hardware with a distributed structure.

  It would be exciting to see a brand new Python implementation to supersede Erlang for next generation concurrent computing !

Best Regards.

[JukkaL]

The current plan actually seems to be pretty close to what you are proposing --- but the roadmap on the web site is not quite up-to-date. The only big thing currently missing from the plans seems to be the DSL part. That's also something I've looked into, but I haven't written about it as I don't have any particular approach in mind yet.

Here are some thoughts:

• Efficient multithreading without the GIL (i.e. not STM) is important for many applications which need shared memory. This is relatively straightforward except for the garbage collector, as mypy has different threading semantics from Python (much less implicit synchronization => more like Java than Python). I'm not sure if anybody has solved the garbage collector issue satisfactorily for allocation-heavy programs; it needs to be something that is actually implementable without a half a dozen PhDs.
• Message-passing concurrency is good for another important class of programs (something like Erlang). IMO, one of the key benefits is the ability to run gc separately in each process/actor without synchronization. Even if copying values between processes is necessary (Python really does not support immutable data well), with high parallelism (say, at least 4 cores) performance is probably significantly better than with multithreading + a typical gc implementation for many applications. Plus, this scales to clusters as well. Copying values between processes can be made more efficient by running multiple mypy processes within a single OS process, as we don't have to serialize objects but can just copy them, similar to how a copying gc works.
• Coroutines may also be a good option for programs that require a very large numbers of concurrent tasks (e.g. many server applications). These can be supported in addition to the above approaches.

I've been planning to write a blog post detailing my current ideas, but I've been too busy recently.

Please ask if you want to know more about the details.

[HowardMei]

Hi Jukkal, Thank you for the update. Your ambition is really adorable. I'm very excited to see these features are in your roadmap. DSL is not a problem if mypy support a flexible parsing/compiling scheme.

I can't agree more with your point 1. It's really hard. A tracing garbage collector like Java can definitely improve the performance at the cost of memory. IMO, the usage of multithreading can be narrowed down for data crunchers only. Let message passing and coroutines handle all other concurrent tasks. Performance of CPU/MEM-bound jobs might also be boosted by using TCMalloc, or the way like http://code.google.com/p/numexpr/wiki/MultiThreadVM.

Anyway, this is really a painful tradeoff. Maybe it's too greedy to ask for built-in solution for low memory embedded system. Also, the introduction of all these concurrency models in native level may inevitably break the compatibility of mypy with python, which I have no idea how important to you. Though I'd like to suggest you let pypy take that legacy burden. Full speed to the future is more enjoyable, bro :-)

It'll be very nice to read your detailed ideas when you have time to post. Best Regards.

[spkersten]

I guess this can be closed?

[JukkaL]

Yeah, this is no longer relevant.