jscl-project / jscl

A Lisp-to-JavaScript compiler bootstrapped from Common Lisp
https://jscl-project.github.io
GNU General Public License v3.0
888 stars 109 forks source link

Developer Documentation & On-boarding #218

Open PuercoPop opened 8 years ago

PuercoPop commented 8 years ago

Looking at the long tail of list of contributors and watchers I think it is fair to say that there are more than few developers interested in contributing to JSCL. We could improve the on-boarding experience by adding a small amount of (developer, not user) documentation. I'm thinking of adding a Doc/Overview.org, improving the Hacking.org file and add a page about medium sized projects in the spirit of SBCL's GSoC page to the wiki.

Overview.org

Adding a couple of questions and maybe a 'TLA'/Glossary file listing some abbreviations used in the codebase (like ll for lambda-list)?

Each project w/ links to information that could be useful for implementing, a link to the issue for discussion and branches with WIP if applicable.

Stuff like Type system, Build process improvements (ej incremental compilation), CLOS, Loop, reader, source maps and javacript pretty printer would go in there

What do you think?

I think I can write about the boostraping process and the AST an submit it on an issue for you to proof read & I'll add some projects to the Wiki tomorrow.

davazp commented 8 years ago

Sounds great!

Do you think it is better doing this in the code when possible (compiler.js, runtime.js (prelude)), or it should go directly in the Wiki or org files?

PuercoPop commented 8 years ago

I think that the org file should give an overview and should not exceed two (3 at most) pages. Something that one can read with ease before diving into the source. Documentation that goes into the details should go in the code (so it is more likely to be updated when modified).

I've added the project page to the wiki, it is still a bit rough, feel free to polish it if needed or add another idea to it.

guicho271828 commented 8 years ago

the page limit is appreciated, or it will become unreadable. Maybe a good material for ILS paper?

PuercoPop commented 8 years ago

This is what I have about the bootstrap process. I am not happy with the 'flow' of the text. it reads liek a list of bullet points. Also I am not sure if the last sentence should be included. But as the saying goes perfect is the enemy of good.

Bootstrap Process
=================

First the compiler is loaded into the host lisp. The compiler now loaded in the
host lisp compiles (`!compile-file`) its own sources along with the
implementation of CL is compiled again to produce `jscl.js`. To ease
development, bootstraping also compiles the tests, a web-repl and
node-repls. The latter compilation also dumps the global environment at the end
of `jscl.js`. The build order is specified in the variable `jscl:*source*` that
can be found in `jscl.lisp`. During the compilation `:js-xc` is present in
\*features\*. To avoid clashing with the host compiler some functions are named
with a precedent bang, ej !proclaim. When the target jscl is compiled the names
are, ej `#+jscl(fset 'proclaim #'!proclaim)`. A function not yet defined can be
used in code as long as it is defined before it is called for the first time.
pnathan commented 8 years ago

IMO - enumerated lists are very good for bootstraps, they give a numerical description of what has to happen, in order.

PuercoPop commented 8 years ago

@pnathan thanks for the suggestion. You were right. I've rewritten it as an ordered list and am much happier with the result.

Bootstrap Process
=================

1. The compiler is loaded into the host lisp.
2. The compiler now loaded in the host lisp compiles (`!compile-file`) its own sources.
3. Along with the resulting JavaScript the environment is serialized  and the result is saved `jscl.js`.
4. Although not strictly part of the bootstrapping itself, the tests, a web-repl and node-REPLs are also compiled and saved as `test.js`, `repl-web.js` and `repl-node.js` respectively.

Notes
=====

- The build order is specified in the variable `jscl:*source*` that can be found in `jscl.lisp`
- During the compilation `:js-xc` is present in \*features\*. 
- To avoid clashing with the host compiler some functions are named with a precedent bang, ej !proclaim. When the target jscl is compiled the names are, ej `#+jscl(fset 'proclaim #'!proclaim)`.
- A function not yet defined can be used in code as long as it is defined before it is called for the first time.
diogoalexandrefranco commented 8 years ago

Hi @PuercoPop, I think this is great.

As a jscl noob trying to understand the project, the most valuable information to have would be everything that you mention in Overview.org. These are all points essential to getting the project but which are difficult to learn just by glancing at the code. I can't think of anything else to add to your overview file, but I'd happily learn about all the points you mentioned.

In Hacking.org, it could also be interesting to have some information about what jscl is still missing or needs improvement (or maybe somewhere else, source code comments/Readme?). Right now, it's only really clear from the Readme and milestone that format, loop and clos are missing. Even if it is too much work to specify and maintain such list with everything in it, just having a few comments at less granularity would be helpful for people getting into jscl, I think. Maybe something like:

sequences

Also, there could be a section somewhere explaining some strategies used in code that you think could confuse jscl beginners. I'm not sure if this is actually worthwhile or if i'm just dumb, but there is some code that I didn't yet figure out, like why we have (defun stringp (s) (stringp s)) and other definitions like this.

davazp commented 8 years ago

I think we can use github issues to keep track of what is missing. I created a ANSI Compliance milestone. Bugs and missing functionality from the standard can be assigned to such milestone so we can easily list them and track progress.

davazp commented 8 years ago

I am on holidays right now but I will write a small document with comments about the compiler and bootstrapping process next week. :)

diogoalexandrefranco commented 8 years ago

Sure, github issues to track missing functionality sounds much better that my suggestion. Should I then start creating issues for every functionality that I find is not yet in accordance with the standard? Like an issue for the count function? Or is that too specific?

davazp commented 8 years ago

it depends. If the functionality is big, led's say type system, it is probably pointless to create all tickets in advance. Instead a "Implement type system" is better. We can create smaller tickets once that issue is started or converted in a milestone itself.

For missing general functions like count, yes, individual issues are preferred I think.

There are a few exceptions, for example implementing Setf for all the c[ad]+r family of functions.

But just go ahead and if someone prefers to break or group tickets we can discuss it there.

PuercoPop commented 8 years ago

@davazp I've started to write some docs on the compiler, they are still far from complete. What can be of most value to you when you are writing the document on the compiler are the questions I'm left with after reading the compiler.lisp file.

= Backend compiler

The backend compiler is composed of two systems.
The first part rewrites s-expressions.
The second part takes s-expressions and generates the corresponding Javascript.

== Rewriting phase

The rewriting phase consists of +define-js-macro+, +define-compilation+, +define-transformation+, +define-raw-builtin+ and +define-builtin+.

+define-js-macro+ is how one defines macros that are going be present in the JSCL run-time (?). Ni idea, parece ser una implementación normal de macros. Tal vez no debería estar como parte del compilador.

+convert+

+define-compilation+ ... | The primitives are stored in +*compilations*+.
+define-transformation+ is similar to +define-compilation+. The difference is that the result is wrapped in a covert.

+define-raw-builtin+ ...
+define-builtin+ works in a similar manner than +define-raw-builtin+.
The only difference is that before executing the +body+, +convert+ is called on every argument.
As they share the storage place (should I use namespace?) the builtins and raw-builtins can nameclash.

The entry point is

See the file 'src/compiler/compiler.lisp' for more details.

=== Questions this section needs to answer

* Is +define-js-macro+ just how macros are implement in jscl or do they have a
  special property that should be noted?

* What is the difference between define-compilation and a normal macro
  definition? Is the only difference that they have access to the environment
  through +*environment*+ variable instead of through the +&environment+
  argument? Corroborate that the +*environment*+ is only the lexical
  environment.

* Why does +with-compilation-enviroment+ not bind the +*environment*+ and
  +*toplevel-compilations*+ variables?

* +define-raw-builtin+ and +define-compilation+ are identicall except for the
  variable where the definitions are stored. What are the differences? Is it on
  the phases they are used?

* Include a Graph following the entrypoint +js+ to each of the functions of
  interest.

== Code generation pharse

See the file 'src/compiler/codegen.lisp' for more details.
PuercoPop commented 8 years ago

@davazp Btw, is there any reason why some docstrings are written as a comment above the function? Or is just a legacy from before docstrings were supported?

davazp commented 8 years ago

Just bad discipline from my side.

I used to write comments for internal functions and do strings for public ones but I didn't stick to it.