Feat/compiler

[happytomatoe]

Improved Jack compiler. Underneath the covers it uses ANTLR List of improvements:

• Added more error messages
• Closes Jack compiler is lagging with big file. Ofc we can speed it up by calling compiler only once. As you can see that 1 key stroke calls compiler 2 times on screenshot.

Cons:

• I've removed the interfaces for built in classes. Though we don't have a validation rule to check if you supplying the right arguments(type) when calling OS classes
• If we want to use advanced ANTLR feature better to switch to JS target

Don't get scared because it says 13k LOC. Most of it are tests and test files. The src changes is around 1k LOC. This doesn't include the generated files.

You can check it out on https://happytomatoe.github.io/web-ide/compiler

TODO:

• [X] Test lexer errors
• [X] Test parser
• [x] Test validations
• [x] Fix error message if class name doesn't match file name
• [x] Validation doesn't work for empty function without return
• [x] Fix bug connected to an error when playing Pong

[happytomatoe]

@DavidSouther @netalondon Could you please take a look?

[happytomatoe]

Upd: I've added validate method to the compiler. Now after each keystroke time of execution is under 100ms for many big files.

[happytomatoe]

qq - why do we write VM files on disk?

[happytomatoe]

@DavidSouther @netalondon just FYI - I've created new repo for jack language server protocol, I would gladly contribute this to nand2tetris repo. About this PR - I would rather have extract compiler into its' own package because of the new repo

[DavidSouther]

Great work in here, I've been following your WIP as it goes.

We're happy to have the LSP in this repo, we just need to spend a bit thinking through where it will go, how to extend it to the other languages, and how to integrate it with the web, cli, and ide extensions.

Again, your use case as a "Jack developer" is relatively niche in the NAND2Tetris community. The vast bulk of work happens at the Hack level, and a little bit at the VM and assembly levels. Improvements to those will have the most impact for NAND2Tetris students.

So finding a migration path from the current state to your ANTLR based approach is a good project for quarter, if someone wanted to pick that up.

Are you ready for me to review this PR?

[happytomatoe]

@DavidSouther Please go ahead. There is a possibility that I can find some small bugs. I can create another PR, if you would like, to fix them.

[happytomatoe]

So finding a migration path from the current state to your ANTLR based approach is a good project for quarter, if someone wanted to pick that up.

@DavidSouther Can you create a new repo? I would provide the scaffolding along with the jack lsp implementation

[DavidSouther]

As you can see that 1 key stroke calls compiler 2 times on screenshot.

In dev, yes; in production, it should still only be called once. That said, we can probably make better use of effects to ensure it happens only once either way.

[happytomatoe]

So finding a migration path from the current state to your ANTLR based approach is a good project for quarter, if someone wanted to pick that up.

@DavidSouther Can you create a new repo? I would provide the scaffolding along with the jack lsp implementation

Nvm. I don't want to spend my time on that.

[DavidSouther]

So finding a migration path from the current state to your ANTLR based approach is a good project for quarter, if someone wanted to pick that up.

@DavidSouther Can you create a new repo? I would provide the scaffolding along with the jack lsp implementation

Nvm. I don't want to spend my time on that.

After we get this compiler in, I think we can look at putting the LSP server in a new folder, server. I prefer monorepo development for contained projects like this that are highly reliant on one another.

[DavidSouther]

@happytomatoe I hope you're able to keep making progress on this, but I wanted to let you know that I'm unavailable for the coming week. I'll be back October 20th to review any changes.

[happytomatoe]

@happytomatoe I hope you're able to keep making progress on this, but I wanted to let you know that I'm unavailable for the coming week. I'll be back October 20th to review any changes.

Have a nice holiday if it's a holiday)

[happytomatoe]

Small update - I was fed up working with official antlr4 typescript target. Migrated to antlr4ng

[DavidSouther]

Small update - I was fed up working with official antlr4 typescript target. Migrated to antlr4ng

Have you looked at Treesitter at all? (question, not a suggestion, but I hear great things about it and might be where I'd start if I was going from scratch today).

[happytomatoe]

Small update - I was fed up working with official antlr4 typescript target. Migrated to antlr4ng

Have you looked at Treesitter at all? (question, not a suggestion, but I hear great things about it and might be where I'd start if I was going from scratch today).

What do you hear specifically? It's a pure C library with language bindings. I didn't work with such combination. Probably it has it's own pain points. I don't know. I've seen this tool when I was building prettier plugin for Jack as it can build syntax tree(Tree with whitespaces and comments.)

If you would build this from scratch, based on my experience, the most go to options are

• Antlr - because it's old and mature. There are commercial companies that use this tool - https://strumenta.com/. There are a lot documentation/resources that can help with niche problems. On the main website looks like Python used this library before switching to own implementation.
• Chevrotain - I've seen java prettier plugin and probably of others prettier plugins that use chevrotain. If comparing with previous option it probably is faster (according to their benchmark). Though we probably don't need a faster lexer/parser. I don't know how mature is this library. I've seen a PR where they are implementing a feature from ANTLR. I also don't know if there are commercial companies that use it.

[happytomatoe]

Testing is done from my subjective point of view.