search

ocaml / merlin

Context sensitive completion for OCaml in Vim and Emacs
https://ocaml.github.io/merlin/
MIT License
1.58k stars 233 forks source link

Advice with custom OCaml syntax. #309

Closed jordwalke closed 9 years ago

jordwalke commented 9 years ago

I'm not sure if there are plans or intentions for merlin to work with ppx extension points, but in addition to ppx extensions, I'm curious if merlin could support a much wider set of custom syntaxes. Could this be possible, and simple, assuming that a custom syntax is capable of mapping its AST to a valid OCaml AST?

let-def commented 9 years ago

PPX extensions will be added.

Supporting custom syntaxes… This was more or less the purpose of camlp4? To provide (relatively) robust parsing, Merlin has its own parser for the OCaml language. Extending the syntax amounts to extending this parser. This can't be done in a "vanilla" merlin so you'll have to fork. A few things could be made to help this process.

jordwalke commented 9 years ago

I may be trivializing things, but to me this sounds like a fairly simple problem to solve. My rationale is the following thought experiment. Imagine transforming from X syntax to valid OCaml from within Vim - and passing that valid OCaml result to merlin. As long as I can construct a "recovered" OCaml AST in the event of X parser errors, and provide the corresponding cursor location in the OCaml AST - shouldn't this be simple?

I'm not saying that this is the best architecture by any means - but it might demonstrate why this could be simpler than I first thought. I'm sure I'm glossing over some of the complexity and I am eager to learn more.

jordwalke commented 9 years ago

I'm curious. Did you fork the OCaml compiler in order to use Menhir (potentially among other reasons)? If so, what was the reasoning for using Menhir? Did you find that it does a better job of error recovery? Should the main OCaml compiler be upgraded to use Menhir?

gasche commented 9 years ago

I'm not a merlin developer, but I can still answer this.

Did you fork the OCaml compiler in order to use Menhir (potentially among other reasons)?

Merlin reuses some code from the OCaml compiler. It's more a "we need a deeper access on some specific parts of the frontend than what compiler-libs currently gives us" thing than a real fork, because Merlin does not act as a compiler and cannot. Yes, re-doing the way OCaml code is parsed is part of the change, the other (and I think originally the more important) are hacking the type-checking code to work relatively well in presence of incomplete code and type errors all over the place.

If so, what was the reasoning for using Menhir?

Merlin is quite compatible with ocamlyacc (so it's easier to start from the compiler's grammar, and then hack your way into one that works for Merlin, and then hack to adapt it to Merlin's need, that is incrementality), strictly nicer to work with than ocamlyacc (which is in C, yuck). Other OCaml parser generators might have been chosen (eg. dypgen which is also a very nice tool).

Did you find that it does a better job of error recovery?

@def-lkb experimented extensively with error recovery and forked Menhir to provide much better support than what can be achieved in any of the off-the-shelf parser generators. I've been communicating with him (and Menhir's authors, François Pottier and Yann Régis-Gianas) about getting some of those features back in upstream Menhir, but these things take time.

Should the main OCaml compiler be upgraded to use Menhir?

Yes, if/when we have a better reason to do so that "it works just as well as before". We need people to do that work, and right now it's unclear when they'll have time for that.

let-def commented 9 years ago

@jordwalke: we could provide a set of tools to easily write what you said (custom parsers generating ocaml AST). This won't be before a loooong time. In the meantime, https://github.com/the-lambda-church/merlin/tree/merlin-ppx branch has PPX support.

jordwalke commented 9 years ago

Glad to see ppx support in progress. Those custom hooks to provide totally custom syntax will be cool, and I agree they're not super high priority right now. But could you at least verify that it's not a totally crazy approach? It seems like even if I were to hack together a solution that only really works in my own case (not suitable for sharing with arbitrary syntaxes), that I could get 80% of the way to a good user experience in a relatively short amount of time.