Features and roadmap - Githubissues

kskyten commented 4 years ago

I'm trying to find an embeddable scheme that can parse Julia and is compatible with the nanopass compiler framework. Upscheme looks very promising. It seems that you have put a fair amount of work into upscheme after forking it from femtolisp. Could you explain the differences to femtolisp in more detail? What are your plans for upscheme going forward?

lassik commented 4 years ago

Thanks a lot for your interest!

Upscheme's goal

Upscheme is meant to be a maximally portable command-line scripting Scheme with a stability guarantee.

Portability

Portability-wise, the platforms listed in the README are already supported.

Upscheme ships as a single, self-contained executable file on all platforms. No support files (Scheme libraries or a boot image) are needed so the executable can be installed in whichever directory is convenient.

As with most Scheme implementations, a significant impediment to portability and smooth builds is bootstrapping (i.e. generating a Scheme image from an earlier Scheme image, which has been generated from an even earlier Scheme image, etc...) Like FemtoLisp, the Upscheme source tree contains a boot image. In Upscheme's case, it's at scheme-boot/boot_image.h. I'd like to get rid of it eventually.

Stability

The stability guarantee will be based on yearly editions of the native API. So instead of (import (upscheme)) you'd be doing (import (upscheme 2020)), (import (upscheme 2021)), etc. Each year's API spec will be frozen after release, with only bug fixes to the implementation being made afterwards. New versions of Upscheme will continue to support prior years' editions of the API so people can be confident that old scripts keep working.

I'll also implement the Upscheme API as libraries for more feature-complete R7RS Schemes, so there is a seamless upgrade path for people who outgrow Upscheme. These libraries will follow the same release pattern of yearly editions.

The Upscheme libraries will be assembled from R7RS and SRFIs as far as possible, eschewing custom APIs. Where I make new APIs, I'll submit SRFIs about them. I don't generally believe there is a point to each Scheme implementation adding custom APIs that aren't available in any of the others. People want portable code - and common APIs are easier to learn, too.

Embedding

Unfortunately for your use case, Upscheme is deliberately designed not to be embeddable. The reason is that embedding complicates the implementation a lot. The entire C codebase would have to be re-entrant, which is usually accomplished by having most functions take a struct upscheme * interpreter state object. Tricky subjects like subprocess spwaning, I/O multiplexing and Unix signal handling are also significantly complicated with embedding. Upscheme will be single-threaded, whereas embeddable interpreters generally need to be thread-safe.

My personal point of view is that embeddable and non-embeddable Scheme interpreters are perhaps best kept separate since their environmental requirements are so different. However, there are already several Scheme interpreters designed for embeddability. In particular:

Chibi-Scheme
TinyScheme and S7 (descendants of the classic pedagogical interpreter Mini-Scheme)
Scheme 9 from Empty Space
Elk

Upscheme implementation - status so far

The changes I have done from Femto-Lisp are mainly:

A very large reorganization, refactoring and modernization of the codebase.
Fixing many portability gotchas.
Changing the bootstrap process so that the Scheme image is now stored as a big binary blob in a single .h file, eliminating the dependency on an external image file at runtime.
New, very portable build scripts - no make required.
Preliminary support for a R7RS-style library system.
A number of new primitive procedures from SRFIs.

Upscheme implementation - future goals

A major goal is to get rid of the boot image, which means we can drop the troublesome bootstrapping process. This means that all primitive Scheme procedures need to be written in C instead of some of them being written in Scheme. I have began moving code from Scheme to C, but found out that C is unfortunately a terrible language for implementing those primitives. To that end, I've been exploring a simple Forth dialect instead (good results so far - Forth is much better for abstraction than C).

I'll also see whether it makes sense to replace the core of the interpreter with a Feeley-Lapalme closure compiler.

This work will take quite a bit more time than I initially expected. Finding a replacement for C has been the main thing that has really slowed down progress. It currently doesn't seem sensible to me to invest more time in improving the C codebase due to C's inherent limitations.

Femto-Lisp maintenance

I started a conservative maintenance fork of Femto-Lisp at https://github.com/lambdaconservatory/femtolisp since Jeff doesn't have time to maintain it anymore. That fork is meant to be consensus-driven, incorporating patches about which there is general agreement among FL users. It's not meant to be my personal vision of FL; if you're interested in working on FL, I'll gladly give you full write access to that repo.