Bootstrapping?

[Ambrevar]

According to the readme, an existing Shen binary is required to build Shen. How can we boostrap Shen then? Is there an older version that can build without Shen?

Boostrapping is important to avoid the Thompson compiler attack.

[tizoc]

Hello @Ambrevar, those bootstrapping instructions are for development. If you go to "releases", you can download any of the shen-cl-vX.Y.Z-sources.tar.gz tarballs which come with all the Shen code precompiled into Lisp. In that case all you need is to have a working Common Lisp compiler.

[Ambrevar]

Ah, thanks, I didn't get that. Maybe the README should be clearer about this.

I ran make sbcl and it produced a ./bin/sbcl/shen binary which works.

Then I got a failure (in the same make command):

./bin/sbcl/shen eval -e "(cd \"kernel/tests\")" -l README.shen -l tests.shen
kernel/tests/

!!! FATAL error: Error opening #P"/home/ambrevar/Downloads/shen-cl-v3.0.3-sources/kernel/tests/README.shen"
Exiting Shen.
make: *** [Makefile:188: test-sbcl] Error 1

[tizoc]

In the case of the "sources" tarball what you want is make build-sbcl, make sbcl tries to run a bunch of tests which are not available in that tarball (those are part of the kernel sources distribution).

I will update the README to make it more explicit how binaries should be built from the precompiled sources release (while it mentions that the precompile step is not required, it is all in a single sentence and not immediately obvious).

[tizoc]

Added instructions here: https://github.com/Shen-Language/shen-cl#building-from-a-precompiled-sources-release

Closing this issue.

[Ambrevar]

Thanks. If you mean the .lsp files in the `compiled' folders then they are not the right way to bootstrap. So how do we bootstrap shen in the end?

[tizoc]

By using the precompiled sources tarball you can easily create a working Shen/SBCL executable (as you did). With a working Shen executable, you can compile .shen code too.

So I'm not sure I'm understanding. What are you missing? can you elaborate?

[Ambrevar]

By "bootstrapping" I mean we can compile Shen fully from source, no binaries / byte code.

This is important for security otherwise your compiler is prone to the Thompson attack:

https://en.wikipedia.org/wiki/Backdoor_(computing)#Compiler_backdoors

See https://www.bootstrappable.org/.

[Ambrevar]

To explain it all in one expression: we must solve the chicken and egg problem :)

[tizoc]

There were many eggs and chickens involved here. The current in-shen implementation of Shen/CL was at some point built by Shen/Scheme, which was itself at first, built from an older version of Shen/Scheme that was implemented in Scheme and ran from sources on chibi-scheme, but was then rewritten in Shen with Chez as a target

The current versions are pre-compiled by Shen/CL itself, but if you decided that trusting the precompiled code is risky, you could use any Shen compiler you want to generate precompiled files yourself.

I checked the examples in your link, and Shen/CL (or any Shen port for that matter) is no different to what I see under the "When an alternative implementation exists" header.

[Ambrevar]

So if I understood you correctly, I need to build schen-scheme 0.15 (which is based on Chibi), and using this schen I can build schen-cl.

If so, I suggest you add this to the documentation, it's important to have bootstrappable languages :)

I checked the examples in your link, and Shen/CL (or any Shen port for that matter) is no different to what I see under the "When an alternative implementation exists" header.

Which header in which link?

[tizoc]

You can use any of the many Shen ports to compile/execute the .shen files in this repository as long as it is reasonably compliant.

Which header in which link?

https://www.bootstrappable.org/best-practises.html#compiler

[Ambrevar]

https://www.bootstrappable.org/best-practises.html#compiler

If all alternatives are non-bootstrappable, then we are stuck with the chicken and egg problem. Shen/CL can only be bootstrapped if at least one other implementation is bootstrappable.

[tizoc]

Only Shen/CL and Shen/Scheme have their compiler code written in Shen.

Two good candidates for you to try are:

• ShenScript is all written in Javascript.
• shen-wasp is based on the old version of Shen/Scheme, and uses the old compiler written in Scheme.

[Ambrevar]

OK, thanks!