implement a proper REPL

[thestinger]

It shouldn't re-compile and re-run the entire input so far with each new addition, and should be able to recover from a compilation error or failure. This should likely be implemented in a separate repository until it's solid and has a working test suite.

A good basis for the design would be the cling REPL for C++ built on top of libclang using LLVM's JIT compiler.

[adnelson]

not sure why this (seemingly) isn't in higher demand. I would love to have a REPL.

[rklaehn]

This would be very nice. When developing scala, I really enjoy trying out language features in the REPL. It does not have to be perfect.

[thestinger]

Posting comments on an issue saying you want it fixed isn't going to make it happen any faster. It just decreases the signal to noise ratio on the bug tracker.

[ghost]

@adnelson, @rklaehn: Rust's test mechanism makes it really easy to run snippets in the same context you'd have in real code. I actually prefer it to ghci and friends. HTH.

[catharsis]

Would it be a good idea to implement this using the existing lexer/parser? I've had a look at mod.rs in libsyntax/parse and it seems to me that it wouldn't be too hard to build something that reads tokens from stdin and emitting tokens as they arrive. Actually, looking at driver.rs (librustc/driver/driver.rs) it looks like this is sort of already possible at a higher level through compile_input, though perhaps that doesn't provide quite the flexibility you'd want in a REPL - not sure.

I'd love to play around with this some (provided I can find the time), but I'd really appreciate if some of the more experienced devs would chime in and warn me off if it's a terrible idea or if there are other plans on how this should be done. Any other hints or suggestions would of course also be highly appreciated.

[abonander]

Could we make use of lli to interpret/JIT LLVM bitcode?

[thestinger]

There's no point in shelling out to a command-line utility when it can be done with the same library functions lli is using. The LLVM JIT is trivial to use, and the only difference relative to AOT compilation is that it's outputting code in-memory and making it callable via linker hacks rather than making an object file. It's not what makes this hard.

[alexchandel]

+1 for ease of adoption. I've learned most languages with the help of an interpreter (python, C#, JS, Haskell all have good ones).

[willprice]

I too agree with alexchandel, is there any work being done on this?

[hastebrot]

I also like to see the missing bindings for LLVMs ExecutionEngine implemented in the rustc crate. It seems that ExecutionEngine is needed for a REPL.

[jauhien]

Yes, ExecutionEngine seems to be needed. Are there any plans to implement full LLVM bindings in rust?

[thestinger]

Implementing a REPL is a complex project, and exposing / using MCJIT is a trivial piece of that puzzle. There's no point of getting side tracked on that issue here.

[japaric]

This issue is a feature request, and needs an approved RFC to be implemented (*). It should be moved to the rust-lang/rfcs repo. (As per the issues policy)

cc @nick29581

(*) I think it makes sense to develop the REPL out of tree (it would still be able to link to librustc, libsyntax or whatever is needed), if it needs more rustc internals/llvm bindings exposed, those bits can be requested via RFCs. (Note that I've no experience writing REPLs/compilers (the closest thing I've worked with are syntax extensions!) so I could be speaking nosense)

[sinistersnare]

I do not think this warrants an RFC because it is not a change to the language or the standard libraries. I am personally OK with keeping A-an-interesting-project labeled issues in this repo.

[nrc]

Yeah, RFCs are only for language features or far-reaching changes to the libs. Leaving here.

[jgmize]

One option for implementing a REPL that I find interesting would be to implement an IPython Kernel for Rust, which would allow you to make use of not only the terminal based frontend, but also other existing IPython (or Jupyter) frontends like the notebook, QT console, or emacs notebook.

[schickling]

:+1: would be a huge gain for the language!

[murarth]

I'm not yet familiar with rustc or LLVM, but I'm interested in trying my hand at this project. I'm looking into MCJIT right now and I've been reading some rustc code. I'm not sure what the problem areas may be, but if anyone has any general advice about this, I would appreciate it.

[thestinger]

MCJIT just allows machine code to be directly generated and run without a temporary file. It's not the right way to get started on this. The hard part is the logic of the REPL itself, and it could just start off by outputting an object file with MCJIT usage added as an unimportant refinement later.

[murarth]

Oh. I thought MCJIT might have been a part of managing persistent memory state of the execution environment. I guess I'm not exactly sure what it does and doesn't do yet. Supposing I compile some object code, how could it be executed with a custom stack and heap (and whatever else may be necessary) controlled by the REPL process?

[thestinger]

MCJIT allows you to generate the machine code for functions in-memory and then call into them. It would be the basis of a solid REPL but it is really only an optimization.

[thestinger]

The same thing can be accomplished by progressively generating object files in a temporary directory and linking them together repeatedly. I think it would be easier to build the REPL around the existing backend and then switch over to MCJIT as an optimization, but of course the person implementing it can do it however they want.

[murarth]

You're probably right about MCJIT. I'm looking exclusively at rustc now. The way I see it, there are three big problems:

• Type- and region-checking whole input with each new round of input.
  Naturally, new functions and expressions will depend on the type information of existing items. This should be relatively easy, but it may require changes to rustc code.
• Incremental code compilation with persistent data.
  Each new round of code will need to know the locations of previously defined stack locals. Also, stack locals must not be dropped at the end of a round of input. I'm not sure how this can be done. I haven't looked into IR and machine code translations yet.
• Dynamic loading and linking crates when user inputs extern crate ....
  Loading crate metadata is already done in the type-checking phase, so this may also be relatively easy.

(Also, if this isn't the place to discuss implementation, I'll gladly take it elsewhere.)

[murarth]

Status update (in case anyone is interested):

• Use of MCJIT through LLVM ExecutionEngine turned out to be rather simple. The old rusti code was making use of that and I used it as a guide. As a result, basic compilation (through librustc) and execution are now functional.
• Loading crates as dynamic libraries was also rather simple. LLVM ExecutionEngine does the job of linking.
• I'm currently looking at the type checker code to see how the aforementioned problem can be solved.
• I hope to release something that will run rounds of input as self-contained functions. It will still be useful as a REPL, though not quite as convenient as one would like.
• I've decided to delay implementation of persistent access to declared locals because it seems like the biggest problem. After something generally useful has been released, I'll give this another look.

[schickling]

Good job @murarth! Really looking forward to see this!

[hastebrot]

Sounds promising, @murarth! Can we use ExecutionEngine directly with Rust (e.g. via src/librustc_llvm/lib.rs) or do we need to add an external LLVM dynamic library?

[jauhien]

@murarth: where is your repo with code? I would like to look at it and may be join you in working on it, if you need any help.

[murarth]

@hastebrot: Some C++ wrapper code in src/rustllvm is necessary for it to work properly with Rust. It defines a few functions that do appear in src/librustc_llvm/lib.rs. Once I've gotten everything working and I'm sure there won't be any more changes necessary, I'll be making a PR with those additions.

@jauhien: I don't have any public code right now. I want to get to a point of minimal working functionality before publishing anything, so it doesn't look like it's broken.

[ghost]

Sounds great! Really looking forward to this.

Eventually, one thing I'd like to have in Rust is some sort of interface that we can build fancier editor modes and tools on top of, where the REPL would be a part of that. I'm thinking something llike what the Idris folks have done with the emacs-mode. They provide an interface (editor agnostic) with some commands which hook into parts of the compiler to provide some very cool functionality.

Would it be within the scope of this project to support such functionality? If not, would it be possible to re-use some of what you're putting together now for something like that?

[murarth]

The above mentioned problems still exist, but I have made a public release with basic functionality. I hope it's not confusing that I call it "rusti" just like the old one and the IRC bot. https://github.com/murarth/rusti

[hastebrot]

Very nice, thanks! For reference, the PR that brings LLVM's ExecutionEngine API to Rust is #19750.

[steveklabnik]

I'm pulling a massive triage effort to get us ready for 1.0. As part of this, I'm moving stuff that's wishlist-like to the RFCs repo, as that's where major new things should get discussed/prioritized.

This issue has been moved to the RFCs repo: rust-lang/rfcs#655