Implement (scheme eval)

[okuoku]

Currently, rapid-scheme have no (direct-)interface for eval and environment. Another headache is (scheme repl).. I don't need any of them for now though.

Tests disabled at https://github.com/okuoku/yuni/commit/b7fea1fd220ea427bee635fbd94acfc906835fbd

[mnieper]

Support for (scheme eval) is a goal for Rapid Scheme. Have ETA on it yet.

(scheme repl) will come later.

[mnieper]

Implementing a usable (scheme eval) is not trivial. I have expressed some possible problems here: https://groups.google.com/forum/#!topic/scheme-reports-wg2/GhMaua0prW4. In order to implement a good (scheme eval), I would like Rapid Scheme to load each library at most once. To make this possible, I plan to divide Rapid Scheme into two parts as follows:

1. The first part is a compiler (much like the current version of Rapid Scheme) that does not yet support the full (scheme eval), but otherwise implements much of the R7RS Scheme language.
2. The second part is a runtime written in that restricted subset (and sharing most of the code with the compiler) which is being compiled using the compiler of 1. and which executes programs in the R7RS Scheme language using eval of the underlying Scheme system.

Using this approach, however, leads to optimal performance only if the host system's eval has good performance. Can Gambit be forced to compile eval'ed code dynamically?

[okuoku]

Can Gambit be forced to compile eval'ed code dynamically?

Nope. Just same as most Scheme->C compiler, Gambit will interpret for evaled code. I think it's okay; I believe most scheme programers do know this fact -- evaled code will be interpreted on statically-compiled implementation -- so we will not try to run performance-intensive code on eval.

Of course, Gambit's compiler and its driver are also available as libraries for Scheme program so I think I can write dynamically compiling eval which will generate .so or .dll on-demand. But I don't think it would be practical (perhaps way too slow for just a eval).

From Gambit's point-of-view, Scheme program is just a loadable set of programs; some of them are statically compiled with C compiler, some of them will be interpreted. So ideally, the frontend should generate loadable core scheme program for each(or chunk of) libraries. Gambit system will take care rest; Gambit compiler can compile loadable program into C and statically compiled into .a or .so, or the Gambit interpreter can interpret it.

Gambit can freely intermix statically-compiled program and interpreted program. For example, global variable defined in loaded program can be referenced on evaled(= interpreted) program.

So, IMHO, the eval alone is not ideal for host Scheme interface. Please consider taking account load to make static compilation work..

[mnieper]

For Rapid Scheme, I would like to retain the possibility that eval'ed code is as first-class as static program code is. This is by far more easier if the static program code is eval'ed as well. Otherwise, the compiler would have to serialize its internal state and the internal state of all statically compiled libraries and would have to embed this information into the binary so that dynamically eval'ed code could be linked together with the statically compiled libraries.

If you don't see how to resolve this problem easily, I think there are two options:

1. Implementing an eval in Gambit that produces shared object files that are dynamically loaded. (After the main program is loaded, this slow eval may be changed into the fast, interpreted one.)
2. For your Gambit backend, you use the compiler that understands mostly everything but eval, and then a custom, not-so-first-class eval is added, which uses the Rapid libraries internally to expand R7RS code.

[okuoku]

Otherwise, the compiler would have to serialize its internal state and the internal state of all statically compiled libraries

This IS required for expanders; SRFI-72 reference implementation(which is used by Larceny and NMosh) and psyntax do this for example. I understand we have considerable caveats on this -- we cannot use records for internal data structure for example.

Anyway, anyhow, this is just a optimization topic. Gambit's eval is (at least for me) still faster than chibi-scheme :) so I think depending on eval performance would work for me.

As an experimental project, Rapid-Gambit does not specifically aim for the performance. Since utilizing backend's horsepower would substantially limit frontend's design, it is up-to-you for choosing design; Rapid-Gambit will do the best within the choice.

[mnieper]

What I would like to implement one day is an eval in Rapid Schene that is backed-up by a JIT compiler using GNU lightning. As GNU lightning assumes a C environment, it should be able to integrate it in the Gambit backend.

[okuoku]

The day means Rapid-scheme became stand-alone implementation of R7RS. While I'm really looking forward to this happen :smile: , but, honestly, I don't see any reason for Rapid-Gambit existence at that point. (Plumbing between JITC and Gambit base system would be too difficult problem -- we need FFI at least, how do we transfer closures etc.)