DOC: language specification

[fhs]

Julia needs a language specification detailing its grammar. For couple of reasons:

1. It'll serve as a documentation for users of the language. I don't think the manual documents everything.
2. People can write compatible parsers for the language.

[StefanKarpinski]

If you want to tackle this, that's ok, but it seems like busy work. Until the language settles down, having formal specifications is just one more thing to be out of sync. Python, Ruby, Perl, Matlab and R don't, afaik, have formal specifications or grammars. Personally, I hate when languages give a formal grammar in their documentation instead of just showing you how things are expressed. It might be good for machines, but it's a terrible way to communicate with humans.

[JeffBezanson]

The parser code itself does a pretty good job of this. For each component of the grammar there is a "parse-x" function that shows what to do. I'm sure we can make it even clearer. I see less value in hand-translating it to some other format that is even less well specified. On Aug 25, 2013 5:33 PM, "Stefan Karpinski" notifications@github.com wrote:

If you want to tackle this, that's ok, but it seems like busy work. Until the language settles down, having formal specifications is just one more thing to be out of sync. Python, Ruby, Perl, Matlab and R don't, afaik, have formal specifications or grammars. Personally, I hate when languages give a formal grammar in their documentation instead of just showing you how things are expressed. It might be good for machines, but it's a terrible way to communicate with humans.

— Reply to this email directly or view it on GitHubhttps://github.com/JuliaLang/julia/issues/4144#issuecomment-23235997 .

[fhs]

Fair enough. I'll try to figure things out from reading the source.

Just FYI, Python and R have language specs (I don't know about others):

• http://docs.python.org/2/reference/
• http://cran.r-project.org/doc/manuals/r-release/R-lang.html

I suspect Julia spec would be much smaller than those, since it's a simpler language.

[JeffBezanson]

True, but our grammar is actually quite fiddly --- for example some contexts are "space sensitive" and some are not, marked by the with-space-sensitive macro. So I think there are good reasons to try hard to make the spec and the implementation the same. We should make sure it is in standard R5RS scheme (right now it isn't quite). Then it can be packaged with libflisp or any other scheme implementation to get a C-callable version, or one of the JS scheme implementations to get a JS-callable version. There are plenty of schemes for the JVM as well. I actually think this is a good solution for getting a fairly universally-usable parser.

[fhs]

I was actually trying to parse julia while preserving comments, and positions of tokens. I don't think the current parser can do that.

If you have the comments and positions, the comments right before a function, etc. can serve as its doc string. You can also pretty print julia source code in a standard format, fixing indentations, etc.

Golang has a parser like that, and it's very useful!

[JeffBezanson]

I think our best bet will be to add those features to the official parser. Obviously others may want to write their own parsers anyway, but if we are going to seriously use these features (e.g. doc strings) we don't want multiple implementations.

[StefanKarpinski]

Producing standardly formatted Julia would be nice, although I think that going from our AST back to formatted code might be a bit tricky since, e.g., we don't distinguish between ternary operators and if-else blocks in AST. This is very handy for manipulating them programmatically but means that we'd need a heuristic for deciding when to use which, although that's probably not too hard – if and else clauses are both single expressions with no assignment.

[cossio]

Would it make sense to revisit the idea of having a more formal but readable specification of the language now that 1.0 is out? Sometimes reading a terse and precise description of an aspect of the language can be more enlightening than an informal description, which seems to be the tone of most of the documentation at the moment.

For example, I really like the organization of Python docs. You have a very accessible Tutorial. Then there is the Language Reference, which contains the formal grammar of every element of the language as well as some explanation in prose.

[StefanKarpinski]

Sure, now would be a good time for someone to work on it if they’re interested. It’s a bit hard to validate that a grammar is correct without it generating a parser (even a slow one) so it might make sense to figure out a way to write a grammar that can actually be used to parse Julia code and check that it works.

[cossio]

Reopen?

[slavchev]

Came here for the same reason. Are there any other resources besides the following two (though they seem enough)?

https://github.com/JuliaLang/julia/blob/v1.0.0/src/julia-syntax.scm https://github.com/JuliaLang/julia/blob/v1.0.0/src/julia-parser.scm

[mwette]

Is the conditional-evaluation construct a statement or an expression? I had to run the program to find out. (Maybe it is in the doc's but I didn't find it there.)

The grammar would be useful for generating {octave,matlab,r} to julia translators.

[StefanKarpinski]

Everything is an expression.

[ghost]

Having something like what Go has https://golang.org/ref/spec will be very useful

[ghost]

why do you guys fear to write a specification?

[StefanKarpinski]

What would the benefit be?

[ghost]

Have you read the Go language's spec? It is readable, detailed and not very long.

Alos having a spec means we can have multiple implementations made by different independent groups.

[KristofferC]

What would the benefit be?

[StefanKarpinski]

having a spec means we can have multiple implementations made by different independent groups.

I'm actively against this: there should be one really excellent open source implementation, not many half-assed implementations.

Have you read the Go language's spec? It is readable, detailed and not very long.

Yes, a mere 30,000 word document. That should be easy enough to produce and keep up-to-date.

That said, as I stated when this was originally requested, I would be fine with there existing a spec if someone wants to write one, I will read it and give feedback. If Julia devs approve it, we can make it official. Hopefully they would also be willing to keep it up-to-date. Otherwise we can make it official only for one release of the language and then if someone else wants and up-to-date version for a newer version of the language, they can update it.

[slavchev]

I'm actively against this: there should be one really excellent open source implementation, not many half-assed implementations.

You are a co-creator of Julia and it is good to hear your opinion. That being said, a good healthy competition might be good for the project. Anyway, it doesn’t seem that there is a major effort in creating other implementations and if I am wrong then may the best one win.

It was a while ago when I was interested in creating some tooling and a language service for Julia so having an official lang spec would make things easier. Accidentally my GitHub notification were set on so I get notifications for these recent comments. Maybe after a month or two I will try to spend some time on revisiting my previous ideas.

[StefanKarpinski]

The problem here is that other people are volunteering us to do work that we think is a waste of time: I don't think a spec is a good way to spend my very limited time and energy. The Go folks felt that it was a good way to spend their effort, which is their call. If someone else wants to write a spec for Julia, they are welcome to do so, and as I said, I would certainly be willing to review it (but not to keep it up to date, that's on them).

Likewise, I believe that having multiple implementations of Julia is a waste of time and energy—mine or anyone else's. If there are people out there who are willing and able to create an implementation of Julia that is so good that it "competes with" the main one, then why the hell aren't they contributing to the main one instead of doing all that pointless work? I'd be legitimately pissed if that happened. Not because of competition but because there's so much work that needs to be done and instead of helping with it, these people are reimplementing what's already been implemented? Why? Julia is already open source and has the most liberal possible license, so there can't possibly be any legal issue. We're also not like CPython which rejects using new compiler technology to make things better or faster. If someone has crazy new compiler ideas to make Julia faster, that's great! Try it out here—we'll help.

[PallHaraldsson]

it doesn’t seem that there is a major effort in creating other implementations

Well, it seems like a LOT of work to me, might be a reasons why not done (there are however all kind of add-on projects, that serve similar purpose).

There were two other implementations, even only forks of Julia, and those are not maintained:

Julia "lite" (I still like to have this, and maybe easy to resurrect): https://github.com/ScottPJones/julia/tree/spj/lite

And Intel's Julia2C was in some sense and implementation (fork), and this: https://discourse.julialang.org/t/status-of-and-need-for-parallelaccelerator/19277

[slavchev]

Firstly, I apologise if I disturbed anyone with my previous comment. I write this for my sake, as I said I haven't used Julia for a long time (maybe 2 years) and I don't speak for the other participants in this thread. From the response I get the impression that the team is really busy and it is natural to get frustrated when a perceived pressure is added from the community. I did OSS and I know it. The resources are never enough.

Also, I really like the project. I've always liked passioned teams. Let me remind you one post I really like.

Why We Created Julia In short, because we are greedy.

This is the passion I talk about. You wanted to create Julia, so you created it. So if there are other people passioned about creating alternative implementation, so be it. They want it and they will create it. Don't feel obliged to be part of it. They ask for a lang spec and they will create one if they want it. You proposed a generous offer the review the spec and I am sure they will appreciate it.

it doesn’t seem that there is a major effort in creating other implementations

well it seems like a LOT of work to me.

I expressed it poorly (I am not a native speaker). What I meant is that it doesn't seem that there are people putting a lot of effort in creating alternative implementations. And to be more precise even if they put a lot of effort it doesn't seem the projects are doing well, at least not being popular. I didn't even know about the "lite" project until you brought it out. And as you said these projects are not maintained and so to speak you validate my intended point.

Finally, although I don't use Julia right now let me thank you and all involved for all hard work.

[ghost]

ok, i understand that julia team has a different approch for documenting the language.

[KristofferC]

Julia "lite" (I still like to have this, and maybe easy to resurrect):

This is definitely not "another implementation" of Julia. It's just Julia with certain parts removed. Doing this does not require a language spec.

[xwaeaewcrhomesysplug]

Why not just take a older version and then create a language specifications. It is still hard but will not affect newer version.

For test implementations wise,the specifications will generally have examples right? Then can take the example include it into the specifications testing. So those who really desperate for specifications can see it,even self verify. Does not actually need other party to test.

It is also like the free pascal or c in the olden days.

Specifications generally have examples,and theory. It however very exhausting to write something like that without financial investment and some assistance.

So if someone were to be rich and kind enough.They may actually help create language specification on at least a specific version. But,honestly....I bet there are plenty companies using it..just......

like yeah golang have it probably because...(for any other reply with golang example)

Just going around finding language with good specifications.Found this thread. Hope this info and insights helped a bit.

[ron-wolf]

Sure, now would be a good time for someone to work on it if they’re interested. It’s a bit hard to validate that a grammar is correct without it generating a parser (even a slow one) so it might make sense to figure out a way to write a grammar that can actually be used to parse Julia code and check that it works.

I am currently taking a course in this and would love to take it on in the near future. I believe generating a parser for verification might be doable using, e.g., Haskell parsing libraries. Megaparsec seems to have good support for many different grammars, which it seems would be necessary for Julia, since I’m not sure if Julia even has a context-free grammar.

[ron-wolf]

Actually, come to think of it that would basically just be a second implementation. Could I get a tip on what kind of parser Julia uses currently, to see if there’s a formal language specification tool that would make a good fit?

[KristofferC]

Could I get a tip on what kind of parser Julia uses currently

You can find it here: https://github.com/JuliaLang/julia/blob/master/src/julia-parser.scm

[mwette]

I did a little work a few years ago to get started on a Julia parser, using my (Guile) Scheme-based parser generator. When I generate parsers, I also generate pretty-printers. The intermediate representation is in SXML (XML using S-expressions). I also have a "mlang" (aka MATLAB, Octave) parser. My goal for the Julia parser was to flush out the pretty-printer in order to generate a MATLAB-to-Julia translator. Check out julia and mlang under https://git.savannah.nongnu.org/cgit/nyacc.git/tree/examples/nyacc/lang. The relevant files to check are "mach.scm" and "pprint.scm".

[ghost]

This doesn't make any sense. You people should learn from New Jersey style. How can you make your language so complicated that you forgot to even write a specification.

[pxeger]

I think this issue needs to distinguish from either:

• standardising the whole language, like Go, JavaScript, C, C++, etc. are
• just providing a formal grammar specification for parsing

This seems to be the source of some of the confusion here

[dsaw]

Adding a grammar specification would make it convenient for anyone to feed it into a parser generator of the language of their choice, so that the AST/IR representation can be directly used in that particular library of their language. For example, a parser in Python would be excellent for conversion of symbolic computational expressions from Julia to Python, where there are powerful libraries like Sympy for symbolic algebra.

[ViralBShah]

Worth mentioning https://github.com/julia-vscode/CSTParser.jl to people who end up here.

[ShalokShalom]

@ViralBShah And is there also a formal grammar, that describes the concrete syntax?

[kaby76]

NB: I may be wrong, as it has been a while since I've last looked for the grammar. There is no formal grammar describing the concrete syntax. The language is simply defined operationally by the code because people write code not specs. The "doc" is just a human trying to reverse engineer what the code does after the fact, and may not actually be correct. The grammar has to be reverse engineered from the source using automated means. Grammars are important because they can be analyzed at a much deeper level than running a performance analysis on the compiler or writing a collection of tests.