Multilingual Federates Branch

[MattEWeber]

I just pushed my proof-of-concept for multilingual federates to the branch multilingual see https://github.com/icyphy/lingua-franca/commit/ae25e5aae9e39e4ddee8db1246b0ea23a4565444. Since I'm not making a pull request for it, I'll post the comment I would written there here as an issue to make sure you all see it. Feel free to close this issue or move this post somewhere else as you see fit.

This is a detailed summary of what it took to get multilingual federates working. You can check out the examples at example/MultilingualTStoC and example/MultilingualCtoTS. Also, there are some important FIXMEs listed at the end.

When I refer to "the target language generator", so far I mean CGenerator and TypeScriptGenerator. I did not change the other target language generators.

When I refer to AST transformations I mean insertGeneratedDelays, makeCommunication, and any potential future functions that modify the Model object.

I added the multiple target to Targets.java. The multiple target has the reserved words of all other targets. I refactored Targets.java, so you don't have duplicate the reserved words arrays for each target generator. This change cleans up some code duplication already in the file where the CCpp target had a duplicate list.

The main challenge is to only perform the following tasks in the target generator if the corresponding component is the same language as the target generator:

• federate instantiations
• federate output files
• reactor class
• reactor instantiations

Consequently, we need a mechanism to know when a federate instantiation, reactor class, or reactor instantiation is in the target language. I chose to implement this in the following way: 1) When MultipleGenerator imports a .lf file, the target for the imported file is recorded and associated with the reactors from that file in the targetByReactor map. This is done in OpenLFImport. 2) Create a deep copy of the Model object for each target language. AST transformations cannot be performed outside of a target language generator because they involve the generation of target language code. But since AST transformations in the target language generator modify the model object (eg. MakeCommunication deletes connections), we need to ensure these modifications in one target language generator don't interfere with the other target language generator. See splitMultilingualModel in MultipleGenerator. 3) Use the targetByReactor map to label the Instantiation and Reactor objects in each Model as "foreign language" when they belong to a different target. Since it didn't seem I could add a field to an Instantiation or a Reactor object without modifying the xtext language definition, I achieved the labeling by modifying the name of a foreign language component with the prefix "__foreignLanguage__". This is safe because all names beginning with "__" are restricted, and since the entire point of the labeling is to ensure the component with the modified name will not be generated. "__foreignLanguage__" will never appear in generated code.

With regard to 3), I considered an alternative implementation. For each target language's model, create a hashMap that knows whether each Instantiation or Reactor object is "foreign language". I chose not to do it this way, because it's less modular. I didn't like the idea of having to pass a separate data structure around the program, especially when it would have to be kept consistant with the Model if the Model were to have any Instantiation or Reactor objects added or modified by an AST transformation.

I disabled AST transformations on the Model object in MultipleGenerator. This is accomplished with the new GeneratorBase field skipASTTransformations which defaults to false.

I gave target generators a new field in their constructors so they may be configured to have RTI generation and compilation disabled. This is important because I chose for RTI generation to be performed by MultipleGenerator. Otherwise we'd get two (or potentially three) RTIs.

In generator base I created an alternative entry point to doGenerate for a target language generator, named generateFromModel. The problem with doGenerate is it directly analyzes the Resource and would ignore the preprocessing from MultipleGenerator. generateFromModel takes preprocessed values as arguments. I refactored the main body of doGenerate in CGenerator and TypeScriptGenertor into a function called CGeneration and typeScriptGeneration respectively so it can be used in both doGenerate and generateFromModel

In GeneratorBase, I created a separate version of analyzeModel, called analyzeModelDirectly to work with generateFromModel. Since analyzeModel analyzes the Resource (the resource containing the source code) to set the Model object, it is incompatible with a preproccessed model. The new function does the same tasks as analyzeModel, but analyzes its Model argument and ignores the resource argument.

In CGenerator's writeCleanCode function, only cleanup code for C language federates.

In GeneratorBase's compileCode function, only compile a federate if it's a singleton (i.e. not part of a federation) or if it's not a foreign language federate.

The makeCommunication function in ASTUtils was particularly challenging to make compatible with multilingual reactors. First, I had to disable the generation of target code for a sender or a receiver when that sender or receiver is foreign language. But then it's also important to make sure the generated code gets typed correctly. If for example the sender is TypeScript and sending a Buffer, but the receiver is C and expecting a char*. This means ensuring the generated actions have the correct types too.

FIXME: Nothing has been implemented to ensure the federated reactor has only target independent code. This is important because the federated reactor gets instantiated in every target, and the generated code almost certainly won't compile if it has foreign language code. I'm not sure how to check for this.

FIXME: There's no mechanism for declaring target language reactors directly written in the main .lf file for the multiple target. There's also no checking done so far to ensure the programmer doesn't try to implement such a reactor.

FIXME: I am currently completely ignoring the target properties of the imported reactors. For each target, I create a target node with the correct name and the keepalive property set to true.

FIXME: Related to the above points, it's unclear to me what should happen if a Multiple target .lf file is imported by another Multiple target .lf file.

[edwardalee]

Cool, thanks Matt. This will be very useful...

Edward

---

Edward A. Lee EECS, UC Berkeley eal@eecs.berkeley.edu http://eecs.berkeley.edu/~eal

On Jul 31, 2020, at 5:36 PM, mew2ub notifications@github.com wrote:

 I just pushed my proof-of-concept for multilingual federates to the branch multilingual. Since I'm not making a pull request for it, I'll post the comment I would written there here as an issue to make sure you all see it. Feel free to close this issue or move this post somewhere else as you see fit.

This is a detailed summary of what it took to get multilingual federates working. You can check out the examples at example/MultilingualTStoC and example/MultilingualCtoTS. Also, there are some important FIXMEs listed at the end.

When I refer to "the target language generator", so far I mean CGenerator and TypeScriptGenerator. I did not change the other target language generators.

When I refer to AST transformations I mean insertGeneratedDelays, makeCommunication, and any potential future functions that modify the Model object.

I added the multiple target to Targets.java. The multiple target has the reserved words of all other targets. I refactored Targets.java, so you don't have duplicate the reserved words arrays for each target generator. This change cleans up some code duplication already in the file where the CCpp target had a duplicate list.

The main challenge is to only perform the following tasks in the target generator if the corresponding component is the same language as the target generator:

federate instantiations federate output files reactor class reactor instantiations Consequently, we need a mechanism to know when a federate instantiation, reactor class, or reactor instantiation is in the target language. I chose to implement this in the following way:

When MultipleGenerator imports a .lf file, the target for the imported file is recorded and associated with the reactors from that file in the targetByReactor map. This is done in OpenLFImport. Create a deep copy of the Model object for each target language. AST transformations cannot be performed outside of a target language generator because they involve the generation of target language code. But since AST transformations in the target language generator modify the model object (eg. MakeCommunication deletes connections), we need to ensure these modifications in one target language generator don't interfere with the other target language generator. See splitMultilingualModel in MultipleGenerator. Use the targetByReactor map to label the Instantiation and Reactor objects in each Model as "foreign language" when they belong to a different target. Since it didn't seem I could add a field to an Instantiation or a Reactor object without modifying the xtext language definition, I achieved the labeling by modifying the name of a foreign language component with the prefix "foreignLanguage". This is safe because all names beginning with "" are restricted, and since the entire point of the labeling is to ensure the component with the modified name will not be generated. "foreignLanguage__" will never appear in generated code. With regard to 3), I considered an alternative implementation. For each target language's model, create a hashMap that knows whether each Instantiation or Reactor object is "foreign language". I chose not to do it this way, because it's less modular. I didn't like the idea of having to pass a separate data structure around the program, especially when it would have to be kept consistant with the Model if the Model were to have any Instantiation or Reactor objects added or modified by an AST transformation.

I disabled AST transformations on the Model object in MultipleGenerator. This is accomplished with the new GeneratorBase field skipASTTransformations which defaults to false.

I gave target generators a new field in their constructors so they may be configured to have RTI generation and compilation disabled. This is important because I chose for RTI generation to be performed by MultipleGenerator. Otherwise we'd get two (or potentially three) RTIs.

In generator base I created an alternative entry point to doGenerate for a target language generator, named generateFromModel. The problem with doGenerate is it directly analyzes the Resource and would ignore the preprocessing from MultipleGenerator. generateFromModel takes preprocessed values as arguments. I refactored the main body of doGenerate in CGenerator and TypeScriptGenertor into a function called CGeneration and typeScriptGeneration respectively so it can be used in both doGenerate and generateFromModel

In GeneratorBase, I created a separate version of analyzeModel, called analyzeModelDirectly to work with generateFromModel. Since analyzeModel analyzes the Resource (the resource containing the source code) to set the Model object, it is incompatible with a preproccessed model. The new function does the same tasks as analyzeModel, but analyzes its Model argument and ignores the resource argument.

In CGenerator's writeCleanCode function, only cleanup code for C language federates.

In GeneratorBase's compileCode function, only compile a federate if it's a singleton (i.e. not part of a federation) or if it's not a foreign language federate.

The makeCommunication function in ASTUtils was particularly challenging to make compatible with multilingual reactors. First, I had to disable the generation of target code for a sender or a receiver when that sender or receiver is foreign language. But then it's also important to make sure the generated code gets typed correctly. If for example the sender is TypeScript and sending a Buffer, but the receiver is C and expecting a char*. This means ensuring the generated actions have the correct types too.

FIXME: Nothing has been implemented to ensure the federated reactor has only target independent code. This is important because the federated reactor gets instantiated in every target, and the generated code almost certainly won't compile if it has foreign language code. I'm not sure how to check for this.

FIXME: There's no mechanism for declaring target language reactors directly written in the main .lf file for the multiple target. There's also no checking done so far to ensure the programmer doesn't try to implement such a reactor.

FIXME: I am currently completely ignoring the target properties of the imported reactors. For each target, I create a target node with the correct name and the keepalive property set to true.

FIXME: Related to the above points, it's unclear to me what should happen if a Multiple target .lf file is imported by another Multiple target .lf file.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.