Technologicat
commented
5 years ago Essay question, so here goes. Maybe I need a blog?
I've briefly considered writing a call graph analyzer for Racket, which is my second favorite language at the moment. So, let's consider it as an example.
At first glance, this should actually be much simpler than for Python, since Lisp syntax is so regular. But Racket being what it is, some other issues arise. First, we must track bindings at different phases. Secondly, while a fully macro-expanded Racket program consists of only 20 or so different forms, if we want to track usage of macros (not only procedures), we can't expand the code before analysis precisely because, by definition, the expander makes the macros go away.
Maybe this is not a problem, if we are interested only in the operator part of each s-expression, i.e. the thing it "calls", whether a procedure or a macro. But what if we want to see what the final expanded code calls? Final at what level of recursive expansion? The source code of the macro itself is of no help - the analyzer would need to understand what part of code is part of the expansion, and what part belongs to the logic of the macro itself. This is the kind of thing you inherently get when you have the whole language there all the time.
Then, Racket is a large language. Modules, generators, exceptions, promises, contracts, pattern matching, two different object systems, the works. There are also the official sister languages lazy/racket and typed/racket. The tagline of the whole project? Solve problems. Make languages. So yeah, it's not really a language, it's a language construction kit. (Yes, you might want one.)
Like any serious Lisp (at least CL), Racket supports customization at the reader level. A good example is sweet-exp, which adds a very pythonic indentation-based (but homoiconic and fully backward compatible) alternative syntax to the Scheme family of Lisps, including Racket. For the specs, see SRFI-105 and SRFI-110. Coming from a Python background, I like sweet-exp a lot, but the Racket community seems to like its parentheses.
(Well, there's the DrRacket IDE, and the support for Racket in Emacs (if that's your go-to IDE) seems pretty good too. If you haven't tried structured editing, it's the right (or even only) tool for handling Lisps. It can help also in other languages. Smartparens works also with Python code. That and rainbow-delimiters, and editing deeply nested parenthesized expressions just became a lot easier. With Spacemacs, Emacs is no longer the usability disaster it used to be.)
In the specific case of sweet-exp, there's the unsweeten utility that can render sweet-exp code back into s-expressions, but there is probably no general solution for arbitrary reader extensions (the whole point of a reader extension being able to change the surface syntax of the language). This may be a problem, since the whole selling point of the Lisp family is its customizability. Lisp is a fluid whereas other programming languages are solids.
So, in all, implementing a call graph analyzer for Racket would perhaps be too much work for something that's currently of minor hobby interest to me ([1], [2]).
If you're really curious, and not yet familiar with Racket, there's an intro in my spring 2018 Python for scientific computing course, see slide sets 10 and 11. And reader customization? Pfft. Real languages let you start at hardware design.
So, back to the main question: Pyan, Python, other languages? The above detour is actually semi-relevant for Python, too - in unpythonic, I pretty much went to town with MacroPy3. And Pydialect adds the plumbing to make Python into a language construction kit.
From my experience, at least macros are a useful technology. But I haven't yet considered the implications of syntactic macros for Pyan.
While macros are not composable (and hence there is an envelope of reachable extensions before the collection of macros becomes too complex for a human to extend), I think they fill an interesting niche: macros can be used for borrowing features from other programming languages. A simple example is Python's "with" in Clojure. True wizards build things like algebraic data types or polymorphic monads.
Another feature Pyan is missing is Cython support. During the last few years I've mostly coded numerics (such as [1], [2]), where a wide-spectrum approach is important, and currently I have no call graph analyzer for Cython code.
Since Cython doesn't change the semantics of calls... much... a good first approximation would be a decythonize function, reading the Cython module and stripping everything that's not standard Python. This would be so we could then feed the code to ast.parse, and then just let Pyan do what it already knows how. This requires a parser, with a grammar for Cython, which is a superset of Python. Probably that could be built in pyparsing, but whether the Cython analysis feature is worth the unavoidable maintenance effort to keep it in sync with the official grammar, I don't know.
As for other mainstream languages, such as C, C++, Java, Fortran - I'm not coding in them anymore (at least not by hand), so they're not that interesting to me. C is probably still the right choice in the systems programming niche, and Fortran in low-level high-performance numerics. For general-purpose programming however, there are better technologies.
In my opinion this includes Python, even though it's already ~30 years old, there's the GIL, and while hacking on Pyan or on unpythonic.syntax, nary a day goes by without cursing at the AST representation. (Q. How many pythons does it take to represent constants right? A. Only one, but it must be the latest version.)
But Python is highly approachable, quick to read and write, quite powerful, and has a huge library ecosystem. And it gets the indentation vs. parentheses issue at least almost right. Maybe for really advanced techniques Racket or Haskell can be better platforms. For anything else, there's Python.
Haskell might be an interesting target, but it's a world of its own, and I'm not that familiar with it.
Julia, a relatively new (~10 years) contender for a high-level language for numerics, is a possible future target, but I'm not making any promises. :)
Uiuran
Just curious if you considered this, at last for matters of software architecture studies.
regards