Open emammoser opened 1 year ago
eposse
commented
1 year ago I've created separate issues for the sub-tasks according to what we discussed, with a change: I've divided the issues into two groups:
(Ignore #454, which I closed as a duplicate of #458.)
In both groups I put the code generation issues at the end, as suggested.
eposse
commented
1 year ago Adding a link to relevant documentation for TT4CCM from ciaox11: Timed Trigger (TT4CCM)
eposse
commented
1 year ago A couple of questions about the component changes to support the new TT4CCM connector. The requirements document that you attached specifies the following:
All component type definitions will have a new checkbox for whether or not they would like to include a TT4CCM connector. This checkbox will be checked by default for any new components or migrated components.
1) When you say "All component type definitions" do you mean "All assembly component type definitions"? I'm wondering if it would make sense to talk about a connector inside a monolithic component.
2) Assuming the new TT4CCM connector type is for assemblies, the idea is that it should be added as a DataSpace element, just like other connector types (i.e. AMI4CCM, CORBA4CCM, PSDD_Event, DDS_Event, DDS_State)?
3) It should be possible to add instances of this TT4CCM connector type as a DataSpace to existing assemblies that do not have it? (I imagine that is the case)
4) I found this IDL definition for TT4CCM. However the IDL doesn't define any connector. Given your later sample IDL to generate, I assume no such IDL connector exists or is required, correct? For other connectors there is a "ConnectorDef" element (e.g. DDS_Event) and we use that element to generate an implementation element in the CDP. Since you ask to generate an implementation element for the new connector, shouldn't we add a ConnectorDef element to the model as well, when creating a component with the TT4CCM connector?
5) I assume we would have a model library just like we have for the other connector types and when we create a component that has the TT4CCM checkbox checked, we would add a "uses" reference to the CCM_TT::TT_Scheduler to the component in the model as well, correct?
6) You also ask to add a connection between the "tt_scheduler" port of a component and the instance of the TT4CCM connector. Your example suggests the component is a monolithic component rather than an assembly. Is this correct? If so, does that mean that we should automatically add a port to the component on creation? And an invisible "CCMConnector" (line) between the port and the TT4CCM DataSpace? Should we create separate connector/data space instances in the same assembly for each component part whose component def was created with the TT4CCM connector checked?
Thanks
emammoser
commented
1 year ago I realize what we are asking for here is a little different than how most things have worked in the past, so we may need a few back-and-forth exchanges here to get on the same page.
First and foremost, nothing that we are aiming for here should be done in assemblies or assembly diagrams. With DDS4CCM or PSDD4CCM connectors, a templated port had to be drawn on the component if we wanted the component to have a specific port, and then a line drawn to a dataspace on an assembly diagram if we wanted the instance for that port to be generated in the CDP file as well as providing a common location for configuration settings to reside and apply to multiple port instances.
This TT4CCM connector is vastly simpler (as far as interface, connection, and middleware implementaiton) than anything we have done before. You could imagine modeling it exactly like DDS4CCM or PSDD4CCM with invisible dataspaces and CCMConnector lines, however, it would be nice if we didn't need to have any actual "invisible" entities cluttering up our model. Our assemblies are already massive today and papyrus struggles to operate on them (saving, generating descriptors, etc..)
A few notes and clarifications:
eposse
commented
1 year ago Hi. Yes, I need further clarification.
Before my follow-up questions, I want to clarify some terminology to ensure we are on the same page.
A Papyrus model, as you know, consists of three files: the .uml file with the abstract structure of the model, the .notation file with the diagrams, and the .di file that identifies them as being the same model. In traditional language technology terminology, the abstract structure is the abstract syntax tree or AST for short. It is the most important part. The diagram is just visual presentation. In Papyrus, you see (part of) the AST in the Model Explorer. Some details are filtered, but one can customize the filters and see most of the underlying structure. On the other hand, the diagrams are shown in the "Papyrus Multi-Diagram Editor".
This is particularly important because everything that we do in the tool, and in particular code generation (IDL, descriptors, etc.) is all driven by the AST. That's what really matters. If something is not in the AST, we cannot generate anything about it. This entails that if we need to generate certain elements in the CDPs, there must be some element(s) in the model that indicate something to be generated. This is the main thing that we need to clarify. What exactly does the model need to have to indicate the generation of the new connector, ports, instances, connections, etc. The question of the checkbox, vs dragging-and-dropping elements, drawing lines, etc. is an orthogonal one. That is a matter of UI. We need to establish first what element(s) need to be in the model to represent TT4CCM connectors, ports, etc. and then we can tackle the question of how the user will tell the tool to put them in the model. So I'll refer to these two issues as "the AST question" and "the UI question" respectively. There is also the "code generation question", about what needs to be generated. But this is one that depends exclusively on the AST, not the UI, and it is largely clear to me. So in the section below about the AST question I am more concerned about what needs to be in the model, rather than what needs to be generated in the CDP and IDL.
Also another clarification: when I suggested to add "invisible" elements, like a port, a dataspace, etc., I meant adding them to the AST, not to the diagrams.
I'll respond and to your enumerated list here, with follow-up questions:
- This TT4CCM connector is not templated by a model defined datatype. It does not change its behavior based on another model defined type.
I understand that the TT4CCM connector is not templated. But there is an IDL for it here. I assume that is the one we are talking about correct?
- It only provides a single interface for the component (tt_scheduler) to use. This interface simply allows for creation and cancellations of timers. To give a little context, we will use this timer to say "Call me back on method in 5 seconds"
If that is the correct IDL, then there are in fact three interfaces (TT_Scheduler, TT_Timer and TT_Handler) as well as one struct (TT_Duration), all of which are needed by the definition of TT_Scheduler. I have created a model library with a CCM_TT module that matches that structure. See the attached view of the Model Explorer with the AST for the library. My idea is that user models would have references to these interfaces (at least the TT_Scheduler interface), for example, making the type of the "tt_scheduler" port to be a reference to TT_Scheduler. Now, I've created this library because, as I said above, if we are going to generate something (both in the CDP and the IDL), we need something in the model that we will be generating from, so the user model would have for example, ports with references to the TT_Scheduler interface. Does this sound like the right path?
- We would like the specification of this connector to be done via. a checkbox when creating the component instead of dragging and dropping a port onto the component, typing the port, creating a dataspace, connecting to the dataspace.
This point is more about the UI, so I'll address it separately below. But it does suggest that you do want the tool to create 1) a typed port, 2) a data space, 3) a connection between the port and the data-space. So my questions here are: a. Is the type of the port supposed to be TT_Scheduler? b. DataSpace elements have a "Connector Type". For example, in BasicPubSub, the only DataSpace has "DDS_Event" as connector type. The "Connector Type" field is a "ConnectorDef" element (whether it's templated or not is separate). In the DDS_DSCP case, "DDS_Event" is a ConnectorDef, as is "DDS_State". In the CCM_AMI case, its "AMI4CCM_Connector". In the CCM_CORBA case, it's "CORBA4CCM_Connector". In the CCM_PSDD case, its "PSDD_Event". However in the CCM_TT IDL there is no connector, only interfaces, so we couldn't have a DataSpace as it would not have any "Connector Type", unless we create a dummy "TT_connector" ConnectorDef in the library for it. Is this what you are suggesting? c. Furthermore, you stated that there should be nothing done in assemblies, so are you suggesting to add the dataspace in the same monolithic component as the one that has the TT_Scheduler port?
- We believe this simplification should work because all TT4CCM connectors will be instantiated with no special config values and since it is just a timer, there is no middleware data passing between these connectors like there is with DDS4CCM and PSDD4CCM
Ok.
- Every component with such a checkbox checked would get a new entry in its IDL for the TT4CCM connector port. As well as a new instance generated in the CDP file as well as a connection between that component and that instance.
I understand. The checkbox is an UI issue, so I'll address it below. As for the codegen issue, it's largely clear to me what needs to be generated from the example in your document. My main questions now are the ones above, about what we need to put in the model itself and where, in order to generate those implementations, instances and connections in the CDP and field in the IDL.
You say that you want users to be able to specify the precence of a TT4CCM connector in a component by a checkbox. The question here is: where? There are three possibilities: 1) the "Create Component" wizard, and 2) the CX tab of the Properties view of a component. 3) Both of the above.
If we go for option 1, then the user will not have the possibility of adding the connector to a component later. It would be added only at creation time, which sounds like it would be too inflexible. So I would suggest option 2 or 3.
A final comment about performance, since you raised the issue regarding saving and code generation. We are very much aware of how slow these are. Saving is itself controlled by Papyrus and EMF, and I'm not sure how much we can do there, but perhaps it could be investigated. As for code generation, the main reason is that the code generators were written a long time ago in a very old pair of languages called XTEND and XPAND which are no longer supported, and are interpreted. I have been looking into migrating them to Xtend2, a much more modern, compiled version of the language which runs much faster. It is doable. But it will take time. If you want, we can create a new issue to make this migration.
emammoser
commented
1 year ago Thanks for the write up above, I think we are converging on an understanding. This might be a good time to point out that our team and our programs are largely oblivious to all of the complexities in the model and really only have the knowledge of what we see in the diagrams and what gets generated. I have been trying to come up to speed on this stuff in the last year to help investigate some of these more intricate issues, but at the end of the day, assuming performance isn't adversely affected, we don't have an opinion of what is needed in the AST provided the IDL and CDP generation is what we expected.
I understand that the TT4CCM connector is not templated. But there is an IDL for it here. I assume that is the one we are talking about correct?
Yes, that is the IDL that we include in our component IDLs to be able to use the TT Scheduler
If that is the correct IDL, then there are in fact three interfaces (TT_Scheduler, TT_Timer and TT_Handler) as well as one struct (TT_Duration), all of which are needed by the definition of TT_Scheduler. I have created a model library with a CCM_TT module that matches that structure. See the attached view of the Model Explorer with the AST for the library. My idea is that user models would have references to these interfaces (at least the TT_Scheduler interface), for example, making the type of the "tt_scheduler" port to be a reference to TT_Scheduler. Now, I've created this library because, as I said above, if we are going to generate something (both in the CDP and the IDL), we need something in the model that we will be generating from, so the user model would have for example, ports with references to the TT_Scheduler interface. Does this sound like the right path?
This is an example of "If the AST needs it, we should add it. But NGC doesn't care". If I look at all of the generated artifacts from the model, both the IDL and the CDP, the only thing that we know is needed is the entrys to the IDL file (Include for the idl you found above and the line specifying that we will use the TT Scheduler) and the entrys in the CDP file (Implementation specification, instance specification, one connection to the component). None of the other interfaces or structs, TT_Timer, TT_Handler, TT_Duration, are exposed about in any of the models generated artifacts. Once our build system generates C++ code from the IDL, we get a ton of code that does use those types, but that is after the modeling stage. I guess what I am trying to say is that if you need to fully define the interfaces in order to create and populate the AST, please do.
This point is more about the UI, so I'll address it separately below. But it does suggest that you do want the tool to create 1) a typed port, 2) a data space, 3) a connection between the port and the data-space. So my questions here are:
We have the same opinion here. If you need to define a typed port, a data space, and a connection between them, then you have to do it. However, One thing about this that does worry me is that adding this many elements to every component's AST add more places for things to go wrong in the future. For example, if a change is made to TT4CCM, we would need a migration (or to use the update templated modules function) to upgrade models.
a. Is the type of the port supposed to be TT_Scheduler?
This seems reasonable. Of course, TT_Scheduler is an interface, not a port, but if the tool doesn't care, neither do we. Everywhere else, we always type ports on components with ports defined in the model. But again, if this works, then this sounds find for us.
b. DataSpace elements have a "Connector Type". For example, in BasicPubSub, the only DataSpace has "DDS_Event" as connector type. The "Connector Type" field is a "ConnectorDef" element (whether it's templated or not is separate). In the DDS_DSCP case, "DDS_Event" is a ConnectorDef, as is "DDS_State". In the CCM_AMI case, its "AMI4CCM_Connector". In the CCM_CORBA case, it's "CORBA4CCM_Connector". In the CCM_PSDD case, its "PSDD_Event". However, in the CCM_TT IDL there is no connector, only interfaces, so we couldn't have a DataSpace as it would not have any "Connector Type", unless we create a dummy "TT_connector" ConnectorDef in the library for it. Is this what you are suggesting? c. Furthermore, you stated that there should be nothing done in assemblies, so are you suggesting to add the dataspace in the same monolithic component as the one that has the TT_Scheduler port?
This question brings me back to wondering if there is another way to keep this simpler instead of trying to force this TT4CCM interface into the existing connector paradigm. Every TT4CCM "DataSpace" (and port/interface) will not require a type, will be exactly the same, and we will want it to be completely hidden from the user.
I am going to suggest something stupid and you can tell me how wrong this thought path was. If every component was given a checkbox for this field, when it comes to generating CDP files, the CX code has an IF statement to see if the checkbox is checked, and if so, it adds the entries to the IDL and CDP file. No extra model library needed (although it wouldn't hurt if one existed), no extra model elements besides some model attribute/field on the component.
If you are of the opinion that we want to avoid specialized and almost hardcoded features like the IF statement suggested above. Maybe we can think of a new avenue to include smaller, simpler interfaces like this that don't involve complicated dataspaces and connections. Maybe a new model element like "Interface Instance" or something that can be typed with TT_Scheduler, added to a component and it is assumed that such a model element doesn't need connections and dataspaces to generate the needed artifacts.
You say that you want users to be able to specify the precence of a TT4CCM connector in a component by a checkbox. The question here is: where? There are three possibilities: 1) the "Create Component" wizard, and 2) the CX tab of the Properties view of a component. 3) Both of the above.
We would opt for 3.
I have been looking into migrating them to Xtend2, a much more modern, compiled version of the language which runs much faster. It is doable. But it will take time. If you want, we can create a new issue to make this migration.
Yes, if you can create a new issue, that would be great if we find ourselves with downtime in the future. Although I think the priority would be lower than many of our current issues and enhancements
Issue and tracking information
Developer's time Estimated effort to fix (hours):
Developer's Actual time spent on fix (hours)
ExFChanges.pdf
Issue reporter to provide a detailed description of the issue in the space below
See PDF for better formatting.
Summary The following changes will be required to support SNA ExF Use-Cases in CX Components automatically including a TT4CCM connector instance. Container Process changes to allow for ExF specification and configuration. Component instance configuration to allow for priority and concurrency specification. All three of the above changes will require CDP changes that will be outlined below. Non-Deployment Planning Changes Component Changes In order to support timer events without the use of a reactor components need to have access to a TT4CCM connector which is implemented and provided by AXCIOMA. This connector should be lightweight enough from a runtime perspective and it works with both ExF and non-ExF deployments. All component type definitions will have a new checkbox for whether or not they would like to include a TT4CCM connector. This checkbox will be checked by default for any new components or migrated components. For components with the checkbox selected, CDP files will reflect this by including the necessary Implementation, instances, component connections and locality constraints for the TT4CCM connections. Artifact Generation Changes Required All components who have the checkbox unchecked will see no changes. All components with the checkbox checked will need the new TT4CCM connector. Note: TT4CCM does not have anything explicitly tied to ExF or non-ExF. It is just a connector that can be used to scheduler timers. It can be used in both ExF or non-ExF containers. That means subsequent sections that focus on toggling ExF settings will also have their tags applied to the below connector settings where appropriate This can be done with the following changes: If at least one TT4CCM connector is needed a new implementation is needed: implementation:
a. i.
a. i.
a. i.
Each component type that has a TT4CCM connector will need an instance of the connector created (Think about how each DDS Port results in an instance) instance: A distinct instance ID will be Required
Each TT4CCM connector instance will need a connection to the component
a. i.
a.
a.
a.
a.
A locality Constraint for each TT4CCM connector will be added
Each component will need an IDL entry for the TT4CCM connector ...
include <ccm_timed_trigger.idl
... module Example { component Component_comp { ... uses CCM_TT::TT_Scheduler tt_scheduler; ... } } Deployment Plan Changes Container Process Changes Container processes will now have a field allowing users to specify if they want the process to be single-threaded(Non-ExF behavior) or multi-threaded (ExF behavior). The default selection for all new container processes or migrated container processes will be single-threaded. If the container process is single-threaded, no changes to the CDP file will be required. If Multi-threaded (ExF) is selected, they can choose how many threads they want to be in their thread pool for that container process with a default of 1. 0 is not allowed. The maximum amount allowed is an unsigned 16 bit integer. To configure a multi-threaded container in the CDP file, the following needs to occur: An ExF Container implementation needs to be specified.
An instance for each container process marked as multi-threaded will be created. Note: the number of threads that the user configures should go into the last config property in the below snippet
a.
a. b.
a. b. c. i.
All components and connectors in a multi-threaded container process will have the container's name specified. The following would be added to each component and connector instance
All connector instances that would be deployed in an ExF container must have a specific implementation artifact for ExF specified. If another instance of the same type was to be deployed to a non ExF container, there would need to be another implementation artifact for the non-ExF implementation. This will probably result in duplicate artifact and implementation tags for connectors that exist in both multi-threaded and single-threaded containers. Also, I have seen that we have duplicate "location" information in implementation tags. The implementation tags call out artifact ID (like below), but then also specify location. I wonder if we can refactor these to remove the artifacts altogether? ExF:
c. i. d. i.
a.
non-ExF
The container needs to be added to the correct locality constraint at the bottom of the CDP
Priority and Concurrency Changes For each component instance deployed, we will be able to specify a priority for the entire component (default 0). Additionally, for each interface on a component that the component provides an implementation for (think on_one_data or any other callback), there will be an option to specify a priority on them. Lastly, we will want the ability to specify a concurrency value for each component (default 1). Something similar to the mockup below:
a. b.
a. b.
Priority To specify a priority on a component or any of the component's ports, a configProperty must be set in the component's instance in the CDP as follows: To set component wide priority
Note: If this is set to 0 then no CDP change needs to be made To set individual port priority
Note: If this is set to 0 (the default) and the component setting is 0 as well, then no CDP change needs to be made Technically, we have the ability to set individual event priority, but we do not believe that we want to attempt this at this time. We will want to have some sort of "?" icon to help explain priorities. The explanation will be something along the lines of "All components and events have a priority of 0, which is the lowest priority. The highest priority is 65,536. If you set a priority other than 0 on a component, all events for that component will inherit that priority. Any priorities set on callback interfaces will override component wide priorities". Concurrency To specify concurrency settings on a component a configProperty must be set in the component's instance in the CDP as follows:
Note: If this is set to 1 (the default), then no setting needs to be made We will want to have some sort of "?" icon to help explain concurrency. The explanation will be something along the lines of "All thread safe components have a concurrency of 1. If this number is increased, you will increase the number of threads allowed to call back into the component, removing its thread safety. The developer will need to ensure thread safety in the component code."