Module punning and use of Core.eval for model construction

[jrevels]

IIUC, Modia currently dynamically instantiates Julia Modules and directly evals code into them in order to construct models. I have some questions here:

• Is my understanding even correct? Any details/code walkthroughs would be appreciated.
• Why was this choice made?
• Given the use of eval...
  • ...how do you ensure proper macro hygiene?
  • ...how do you ensure proper error/type/invalidation checking?
  • ...is the cost of model construction just always negligible and not worth optimizing?

Executing code via eval is essentially saying "dynamically evaluate the given Expr at top-level within the given Module". From the perspective of a local callsite, the Julia compiler can assume very little about what an eval invocation will do to the state of the runtime. It certainly can't optimize through an eval invocation, and it can't even know, for example, whether or not locally valid definitions of top-level constructs (e.g. methods) will be invalidated by the eval invocation (which is why invokelatest has to exist).

AFAIK, no other Julia-based modeling language takes this Module/eval approach. Instead, most Julia-based modeling languages represent their programs via custom data structures (if the language has separate semantics from Julia itself) and/or via normal Julia functions (if the language's semantics are a near-superset of Julia's). Some DSLs utilize macro-based front-ends, performing extra syntax checking at parse time (within the macro) and/or at runtime (via type propagation in the generated code), while others try to extract a reasonable program representation directly from Julia code via operator overloading/tracing/source code analysis/etc.

With so many potential strategies that already have found success in the Julia ecosystem, what is the advantage of a strategy that relies so heavily on traditionally dangerous escape hatches like eval?

It is certainly possible that evaling into dynamically constructed Modules is actually the appropriate design for some task; for example, it's what we do in Cassette, but that's to handle extremely specific situations where Cassette is essentially interacting directly with the Julia compiler/runtime. It doesn't seem like the same thing is going on here, though I could be mistaken.

[toivoh]

Hi, Those are some excellent questions, and I'll try to answer what I can.

Modia does use eval, but only as a last step, when it should have pretty good control over what is in that AST. I'll try to outline the route that a model takes from entry to execution, it's somewhat complex.

I hope this gives a bit of an overview, and starts to answer some of the questions. I'm sure that it doesn't answer all of them, but for those that it doesn't, could you restate them given the overview above? I think that would make things easier. Also, I hope I'm making some sense, please ask if something is unclear.

The @model macro

The @model macro takes a textual description of a model as input. When the code produced by the macro is run, the result is to assign a Model instance to the variable that has the same name as the model. E.g.

@model M begin
    ...
end

==>

M = Model( ... )

The Model is basically contains a set of instructions for how fill out an Instance. In the Instance, equations are represented by ASTs, while variable bindings are represented by a dict with (name, Julia object) pairs. If the object bound to name is an

• Instance, it represents a sub-model,
• Variable, it represents a variable. Other types are interpreted as constant values at the moment. The Model representation also needs to be independent of the point in the code where it was defined.

The @model macro does not use Julia's macro hygiene, because it was found too limiting (you can see that it wraps the whole return AST in an esc()). Instead, it implements manual hygiene by not inserting any symbols into the AST that weren't there to begin with (it interpolates values into the AST instead).

The rewriting done by the macro is mostly located to the function recode(ex). This takes an AST and rewrites it into an AST that will evaluate into a version of the original AST where e.g. all symbols have been evaluated and replaced by their bindings. It also creates special representations for e.g.

• der() (Der <: Symbolic)
• connect() (Connect <: Symbolic)

• dotted access (GetField <: Symbolic, through model_getfield(x, name), if x is Symbolic) There is also This <: Symbolic, that represents the current instance. Initially, all variables in a model had to be access through this, e.g.

  der(this.x) = -this.x

but to ease familiarity for Modelica users, a number of shortcuts have been implemented to insert the this. in front of the x.

So when the Model object is constructed, the ASTs inside are considered to be clean. I think that they might as well be considered Modia-internal data structures, which is reinforced by the fact that they contain some Symbolic objects mixed in.

Initializers for variables in the model are not passed through recode(). Instead, the code of each is wrapped into a function that is also evaluated as part of the code created by the @model macro, and saved in an intializer in the Model.

Instantiation

The model is instantiated through instantiate(model::Model, time::Float64, kwargs). This will recursively intantiate any models nested as component of the top model, and results in an instance tree of nested Intstances. The saved initializer functions are run, and the saved equations are copied into the Instances.

Flattening

Flattening takes an instance tree and rewrites into one big flat instance without nested instances. Variable names are rewritten to include literal dots in their names to signify where they have come from. Connection sets are also resolved in this step.

Code generation

This stage takes a flat instance and generates a residual function that can be passed to a numerical solver. It

• substitutes actual variable names instead of e.g. this.M.x and der(this.y).
• detects differential states (variables that appear inside a der())

This is where the eval is used. But it's not used on raw ASTs, the hygiene has already been taken care of by the @model Macro. (I'd be interested to hear if you think there's cases that are not covered) The eval is actually used twice:

• Once, two evaluate everything at the initial time. The result types are used to infer e.g. the dimensionality of each equation.
• Finally, to create the callback function. The callback function includes code to unpack variables from the state vector passed in from the caller, and pack residuals into the residual vector that is the output.