Modules

[cardillan]

A draft for adding support for modules to Mindcode. To be implemented in 2 phases.

NOTE: parameters in this text aren't function parameters, but global variables declared using the param keyword and used to allow parametrization of the compiled mlog code.

1st phase - Support for loading additional files

• require keyword for loading other source files. Takes a filename as an argument (require "../library/system.mnd").
• Each loaded file would be parsed separately. After parsing the topmost file and all loaded files, the parse trees would be combined.
  • Loads would be handled recursively. File which was already processed wouldn't be processed again --> cycles in module dependencies are allowed.
    • Paths are resolved relative to the file containing the require declaration (is this optimal?)
    • File paths would be converted to canonical paths to identify already processed files.
    • Given the inherently limited size od Mindcode projects, I don't expect complicated dependencies between modules.
  • Syntax errors will be reported per file.
• The entrypoint (begin ... end or alternatively def main()), if present, must be contained in the main file.
• Rules for combining parse trees:
  • Parse trees are combined at the topmost level only (topmost nodes merged together).
    • AST contexts assigned to the nodes will contain information about the source file and module
  • allocate stack and allocate heap, if declared multiple times, must be declared identically.
  • A function must be declared only once.
  • A constant or parameter can be declared multiple times if the assigned value is always the same.
    • Perhaps a constant or parameter in a module might be declared without initial value, which would be provided by the main module.
    • Alternatively, value assigned by a module can be reassigned by the main module.
  • A global array/variable can be declared multiple times either with initial value or without it. The initial values, if assigned, must be the same.
  • The combined parse tree will be compiled/optimized.
    • Optimizations run once against the entire compiled code.
    • Everything will be properly resolved, no compilation against uncertain/unknown constructs.

2nd phase - Modules, namespaces and a system library

• Modules
  • Module names specified using module keyword: module foo;
    • A Java/C# convention for module names - identifiers separated by a dot: module cardillan.drawing.icons;
  • module needs to be the first declaration in the file.
  • The first loaded file contains the main module. Main module has no name (no module declaration). Main module cannot be required.
  • Each module has its own namespace named after the module identifier. The main module uses global namespace.
    • A command to import default objects from a given module into the global namespace
      • import foo.bar; - imports just one identifier from foo
      • import foo.*; - imports all
  • Visibility: public/private/default - needs a lot of thought!
    • A public object is always included in the global namespace (parameters are always public)
      • Probably not allowed for functions - avoid global namespace pollution.
    • A private object is inaccessible outside the module (preventing the access is additional work, possibly in a 3rd phase)
    • A default object can be accessed via full module name or imported into the global namespace using import
  • No support for splitting modules into several files until really needed, hopefully never
    • The same module declared in two different loaded files causes an error
• Compilation of modules without a main module
  • A module can be compiled separately.
  • If the module doesn't contain an entrypoint, all functions would be considered reachable and wouldn't be removed by unreachable code elimination.
  • If all dependencies of the module are resolved, a valid, optimized mlog code would be produced
    • Easy verification the module actually compiles
    • Generated code may be inspected
• Target Mindcode versions
  • Modules may indicate a range of Mindustry Logic versions they support.
    • #target ver; or #target ver1 .. ver2; (trying not to allocate new keywords for this).
    • The way of labeling the Mindustry Logic versions has not been specified yet, maybe using build numbers.
    • When not specified, all versions must be supported.
  • Loading a module incompatible with the current target results into an error
• System modules
  • System modules will contain a system library of functions.
  • System modules will reside in GitHub Mindcode repository and will be implicitly accessible, even from the web app
    • Compiled on each build to make sure they work, using the latest supported Mindustry Logic version specified by the module as the target
    • System modules would be required by module names, not by file names.
      • require standard; or require cardillan.drawing.icons; - no double quotes
      • Source files for these modules would be located and imported automatically based on module name, using a config file automatically generated during the test build assigning source files to module names.
      • Parse trees of system modules generated during the test build could be stored to save time - would make sense if we build a really, really large collection of system functions.
        • Unused functions from such a large parse tree wouldn't be even compiled - no time wasted on them.
    • Contributions to system modules would need to be carefully vetted
    • Module standard: system module for standard library functions (e.g. findUnit)
    • Module wp to enclose world processor instructions.
      • import wp.*; would be required, unless we make an exception and allow public functions in module wp (or in all system modules).
      • No more standard/world targets.
  • Host other modules ("user modules"?) in the repository as contributions from different users
    • Included via pull requests
      • No vetting process, except that they compile
      • If they fail to compile, they'll be removed from the release - probably via some config file listing defunct user modules
    • May need establishing a naming convention for user modules, perhaps after github usernames (e.g. cardillan.buildings, cardillan.drawings etc.)
    • Instead of require cardillan.drawing.icons use require "cardillan/drawing/icons.mnd" to develop against a local version of a module.

Example:

require standard;

begin
    unit = standard.findUnit(@mega);
    // Do something with unit
end;

Importing two global variables might merge them - do we want to allow this? Might be useful in some situations - e.g. if two modules want to share variables referencing to units.

module foo;
var debug;
...

module bar;
var debug;          // Variables not public - this is different from foo.debug!

require foo, bar;
import foo.debug
import bar.debug;   // Merges foo.debug with bar.debug. Or error. 

Rejected ideas

• Replace language targets with modules: the program would include the module for the proper target
  • What if two modules include conflicting target modules?
• Support for different targets in one module: have sections of code only be included when compiling for given target. That way the same logic could be implemented using instruction sets from different targets.
  • Support for different Mindustry versions doesn't seem to be crucial.
  • Can be still made by moving target-dependent logic into separate modules and including them as necessary.
• Using C-like #include directive.
  • The included file can be put anywhere - say, inside a loop. That means it generally cannot be parsed separately.
  • There's no direct support from ANTLR to do this, so it means either using/writing a preprocessor, or finding a correct way for ANTLR To do it. All seem to be a lot of work.
  • Line numbers would have to be fixed when reporting syntax errors.
• Compiled libraries
  • Each source would compile separately and the resulting code would be combined (linked) somehow
  • Lots and lots of work:
    • Needs a linker
    • Every file would be optimized separately - a global optimization step might be required after linking
  • Mindcode will never be used for such large projects that separate compilation of modules could save some significant time.

[cardillan]

The first phase has been more or less completed as of release 2.6.0. On to of that, system libraries were also implemented.

What remains to be done is namespaces and modules.