Modules Forever

[JohnDTill]

So the code aspects are completed up to per-file symbol resolution. I think there is a small step to gather information for project symbol usage for the sake of the IDE, and a big step to revise the static typing pass. The IDE changes are mostly independent of the code changes at this point.

• [x] Revise IDE handling of files
  • [x] Handle main file consistently with imported files
  • [x] Revise scanning for changes to support the whole project
  • [x] Implement the project browser
  • [x] Find common ancestor directory
  • [x] Display information about the currently viewed file correctly
  • [x] Cache the currently viewed file and the program entry point
  • [x] Decide when to set IDE back/forward jump points
  • [x] error/search clickable links
• [ ] IDE features work across files
  • [ ] Decide data structure for finding symbols used in other files
  • [ ] Find usages
  • [ ] Go to def
  • [ ] Rename
• [ ] Rewrite static pass to take disparate files (with parse trees and symbol tables) and produce a single AST
• [ ] Make sure IDE information is preserved (mainly hover for type / value)

So there's a lot there, but this process has stumbled onto an elegant design, and the steps are relatively self contained.

Deferred:

• search/replace for multiple files. Very useful, but little doubt it can be implemented and not critical.
• File names to go with link outputs. This should be revised to present the right information, but no rush

[JohnDTill]

From last time, the IDE problems for multifile support can be solved relatively independently of the code by using simple include statements. That includes:

• Making the symbol table searchable despite being built from multiple files
• Implementing the file browser
• Some kind of scheme for setting the program entry point file (while still going off to edit other files)
• Context click options to follow file paths in the editor
• Decisions for if a file can be included multiple times, how the syntax highlighting if so
• Caching loaded files and minimising I/O

[JohnDTill]

Include statements are a hack. A useful hack, like macros‡, but a hack. It would behoove you to focus on modules first.

You need to declare anything you do with modules†, so cyclical references aren't a difficult issue. As I think about it, Forscape's current decision to do classic simultaneous declaration and reference resolution is not a problem for modules. It might be worth revisiting in its own right to allow codependency without prototypes, but that's not necessary for modules since you already need to declare which modules you are using.

Modules being decoupled as described above, the current file handling in the scanner is a bit daft. You really can parse the files one at a time, which makes figuring out the multi-file symbol table data structure much easier. Perhaps each document could have it's own symbol table data structure. If each file has it's own parse tree and symbol table, then I think you're finally starting to see the path towards partial compilation.

†The only caveat is that you can't do a "from MyModule import " without a two-pass symbol resolution, but that capability isn't central to modules. In fact, you would need to be utterly convinced that "import " capability is extremely valuable to the user before breaking the modularity of modules... and I don't think that will happen.

‡In C/C++ inclusions are quite similar to macros in that they substitute code, only from a text file instead of directly inline. You could even have "arguments" to an include statement using macro definitions. The preprocessor is a great and terrible thing...

[JohnDTill]

So how does the typing work? In the decoupled modules, you could pass symbol resolution with

moduleA{ var = moduleB.var }
moduleB{ var = moduleA.var }

Especially since you want to allow generic functions without type annotations, the scope of a typing pass is the whole program, probably starting at the program entry point. Probably this would be a good time to rework the static pass and use it to create a new AST instead of marking up the parse tree.

[JohnDTill]

There are currently 4 ways to reference symbols across files:

1. import module then module.var
2. import module as m then m.var
3. from module import var then directly reference var
4. from module import var as v then reference v

Classes will add more ways, with the caveat that you can use the . operator on an object to do an access at a known offset of the object.

So you can think about the data structure to store and update inter-file symbol references to be used in IDE interaction, but populating that information will fundmentally rely on the static pass. It is not a small operation before the static pass as I previously thought.

[JohnDTill]

So there's a good, modular design, and the frontend work is DONE. Now the only small task left is a total rewrite of the backend. I should probably merge the frontend work before yeeting the backend...