New compiler

[tizoc]

Still have to cleanup the bootstrapping process.

TODO:

• [x] cleanup new bootstrapping process
• [x] update Makefile
• [x] update travis

Some notes:

• Test suite doesn't run that much faster because right now the compiler itself is slower than the older one (because it does more)
• Added overrides for dict functions.
• Considerably better code generated, in particular dynamic function applications is much faster now and doesn't allocate anything (the old compiler produced code that resulted in many unnecessary allocations). When testing the factorisation benchmark, it outperformed Shen/Scheme.
• When generating .lsp files all symbols that have to preserve their case are wrapped in |s, which means it may be possible to get rid of the read-table modifications.
• Bootstrapping will look quite different after this is all done, quite similar to how Shen/Scheme's bootstrapping process works. It can also be bootstrapped using another Shen.

[rkoeninger]

Compiler looks great and those unit tests are a good addition!

How does this relate to backend.shen and backend-boostrap.lsp? I take it they can be removed? Does this mean shen.exe is required for building now? Or will there be a compiler-bootstrap.lsp?

[tizoc]

@rkoeninger backend* goes away. A working Shen compiler will be needed to bootstrap Shen/CL from the repository (any compliant Shen should work, because no platform specific extensions are used -- I have been using Shen/Scheme to make sure it works), but source releases will include all the precompiled files, so in that case one of the supported Common Lisp implementations will be all one needs to build Shen/CL.

[tizoc]

Ok, this is almost ready for merging, and warrants a 3.0 release.

Something that is not working right now is that the new compiler cannot compiler itself, still trying to figure out why it is failing (it is related to how it generates factorised defuns, the NIL in the block expression is being quoted, but I don't understand why, and only happens when compiling the code using this version of the compiler).

[tizoc]

Ok, fixed that.

[tizoc]

I guess I should also lowercase the Common Lisp code too, upper-case is not necessary anymore.

[rkoeninger]

Force eol=lf, breaking build on travis

I don't remember line endings being an issue before. What changed?

[tizoc]

The build script extracts the license from the files it compiles instead of hardcoding it.

Edit: to elaborate bit more, the files are parsed as a list of bytes, and since git on windows changes the line endings, that breaks.

[tizoc]

@rkoeninger do you know if Windows (or the CL ports on Windows) do some automatic conversion from "\n" to "\r\n" when writing to a stream? Because this made me check the nl function in the kernel and it only writes the byte 10 (\n), not 13 + 10 (\r\n).

[rkoeninger]

I don't believe CL on Windows (and definitely not Windows itself) adds the "\r". CL on Windows will probably use "\r\n" when writing "~%" as part of a FORMAT.

nl only does "\n" but i think the parser handles both?

[tizoc]

Yes, the reader doesn't mind \r, because it doesn't mind newlines at all, they are just whitespace.

[tizoc]

@rkoeninger I updated the build script so that it handles CRLF and reverted the "eol=lf" attribute change.