search

modelica / ModelicaSpecification

Specification of the Modelica Language
https://specification.modelica.org
Creative Commons Attribution Share Alike 4.0 International
102 stars 41 forks source link

Package improvements #573

Closed modelica-trac-importer closed 3 years ago

modelica-trac-importer commented 6 years ago

Modified by dietmarw on 22 Jun 2011 06:58 UTC

Description

The task is to improve the package concept, there is a summary in: https://svn.modelica.org/projects/ModelicaDesign/trunk/MeetingMinutesMaterial/min71/Slides-and-Documents/Slides-Johan-Andreasson-LibraryManagement.pdf At the 71st design meeting, this was presented and the topic was discussed as described in: https://svn.modelica.org/projects/ModelicaDesign/trunk/MeetingMinutesMaterial/min71/Slides-and-Documents/Minutes-package-concept.pdf Similar previous discussions are available in: https://svn.modelica.org/projects/ModelicaDesign/trunk/MeetingMinutesMaterial/min71/Slides-and-Documents/Improving-Packages-FromOldMinutes/

Action Items

Reported by johan on 20 Jun 2011 10:12 UTC

Description

The task is to improve the package concept, there is a summary in: https://svn.modelica.org/projects/ModelicaDesign/trunk/MeetingMinutesMaterial/min71/Slides-and-Documents/Slides-Johan-Andreasson-LibraryManagement.pdf At the 71st design meeting, this was presented and the topic was discussed as described in: https://svn.modelica.org/projects/ModelicaDesign/trunk/MeetingMinutesMaterial/min71/Slides-and-Documents/Minutes-package-concept.pdf Similar previous discussions are available in: https://svn.modelica.org/projects/ModelicaDesign/trunk/MeetingMinutesMaterial/min71/Slides-and-Documents/Improving-Packages-FromOldMinutes/

Action Items

Migrated-From: https://trac.modelica.org/Modelica/ticket/573

modelica-trac-importer commented 6 years ago

Modified by dietmarw on 22 Jun 2011 06:58 UTC

modelica-trac-importer commented 6 years ago

Comment by johan on 23 Jun 2011 11:17 UTC All modelicans invited via mail, joined so far: OLSSON Hans Martin Otter Michael Tiller Mike Dempsey Peter Harman Dietmar Winkler Gerd Kurzbach (invited) Stefan Vorkoetter (invited)

Poll for first meeting: http://doodle.com/qen8m7ugmsc32ha4

modelica-trac-importer commented 6 years ago

Comment by mtiller on 8 Aug 2011 09:33 UTC I wanted to add some comments here since I'm not sure the "minutes" we have here really explain our proposals from the 71st design meeting very well. There, we struck with the details, but didn't give a very good "big picture" summary.

Currently, the main issue with Modelica packages is that they are monolithic. A single version number is associated with the entire library and it is very difficult to work with it at a finer level. In contrast, in Java every package is a distinct stand-alone entity. The naming has nothing to do with lookup. Only classes in the same package can be looked up without an import or fully qualified name.

What we want is a balance between these two extremes. For example, does it make sense that Modelica.Mechanics.Translational.Components should be completely separate from Modelica.Mechanics.Translational.Sensors? Probably not. So (with one important exception I will get to later), it seems reasonable to break the Modelica library into a set of libraries where each library represents a sub-tree of the current library.

Each of these new (sub-tree) libraries would be required to be encapsulated and have a distinct version number. In fact, the "version number" of the Modelica library would no longer be specified in the Modelica language. It would only be a convention associated with each release. For example, version "3.4" of the Modelica standard library might simply be a bundle of the following libraries (all starting with Modelica.):

Now, when Modelica "3.4.1" is released, it might have a manifest like this:

The fact that each library gets their own version number is very useful because it doesn't mean artificially incrementing version numbers unnecessarily. In fact, this leads to another important aspect of the design proposal from the 71st design meeting, nested versioned packages. For example, it should be (alternatively) possible for version "3.4" of the Modelica library to be composed of the following:

Why? Because when version "3.4.1" comes out, it might look like this:

So user models that only depended on connector definitions don't need to be migrated! Connectors rarely change and this should be reflected in the way things are versioned to avoid unnecessary migrations. This also means that a given model can work with multiple versions of the Modelica standard library as long as the version number of the sub-libraries that it actually uses don't change. So a model that uses Modelica.Mechanics.Rotational.Interfaces version 3.4.0 could work the Modelica library version "3.4" and "3.4.1" because the version number of the Modelica library is just a naming convention to refer to a specific set of sub-library versions.

But there is more. It may be that an interfaces library version number does change but the changes were to connectors that we don't use. In that case, it should be possible to express the compatibility with libraries as was proposed in the library distribution proposal, i.e. providing sets or ranges of versions that are known to work. This will also eliminate the need to "migrate" when it isn't really necessary. So the proposal from the 71st includes a provision to allow the uses annotation to include multiple version numbers.

Finally, the new proposal allows us to overlay new versions of libraries on top of older ones for development purposes. Peter Harman pointed out that for ModelicaServices it might be useful for tool vendors to override part but not all of the standard services library. Also, as standard library developers update their libraries, they can overlay them on top of previous distributions. This is possible because the each library is loaded separately in the MODELICAPATH. For example, for simplicity's sake consider if the Modelica version "3.4" distribution included only:

...the proposal from the 71st design meeting would indicate a directory structure as follows:

and the Modelica directory would be in the MODELICAPATH. However, if the developer of Modelica.Mechanics.Translational wanted to develop their own version, they could check it out into a directory as follows:

By adding DeveloperDirectory to MODELICAPATH ahead of the Modelica directory, version 3.4.0.1 would take precedence over version 3.4.0. The developer could modify that version of the library independently of the others and even release new versions of the library to users without the rest of the Modelica libraries as long as the user had the other required versions somewhere in their MODELICAPATH.

Surprisingly, when we reviewed the specification, there was very little that prohibited these different semantics for processing packages. Basically, we had to allow versioning of libraries that were not top-level packages and we had to change lookup rules so that "structured entities" that were not top-level could appear anywhere in the MODELICAPATH (which is pretty straightforward assuming a within clause is present). There some implicit assumptions (not specified) that all tools seem to follow that will also have to be changed but our emphasis in our proposal was on making the language specification slightly more general so that it did not prohibit what we wanted to do. Those implicit assumptions will also need to be changed but since they are not in the specification (yet), we could not address those with our proposal. Assuming the basic premise of our proposal is accepted, we further propose that the wording of the specification be made a bit more explicitly about how package loading actually occurs to avoid any confusion in the future.

modelica-trac-importer commented 6 years ago

Comment by mtiller on 9 Aug 2011 07:47 UTC Sorry, too much copying and pasting (of erroneous code) above. All the within clauses are wrong. But the basic idea is that individual libraries don't need to be stored in deeply nested directories or even within the Modelica hierarchy at all. The only thing that is required is that the structured entity (directory of file name) be a fully qualified name (if it is not a top-level package) and that it contain the appropriate within clause to indicate its parent.

modelica-trac-importer commented 6 years ago

Comment by mtiller on 22 Aug 2011 08:04 UTC At this point, I am aware of two outstanding issues (please point out any others).

First, Christoph Höger pointed out that "clients" (models that use libraries) cannot know what versions of those libraries they are compatible with. So he suggested that instead each client specify exactly one version of the library in its uses annotation and that the libraries include some kind of backward compatibility information (i.e. the library would have a way of indicating "models that worked with 3.2.0, should also work with 3.2.1"). The logic here is clear, the library is in a better position to know what changes have been made and to state that in exactly one place definitively. However, there is a slight wrinkle here because a library is composed of N different models but the client likely only uses M of those models where M << N. As such, even in cases where the library is not strictly backward compatible for any client, it may still be backward compatible for a specific client. Any suggestions on how to set up the dependency information to be simple, robust and (at the same time) minimize the amount of unnecessary migration?

The second issue is something I noticed while reviewing the minutes. At the time, we determined it was "safe" to allow us to extend an existing package hierarchy if two conditions were met. The first was that the package being added was encapsulated and the second was that all of its sibling packages were also encapsulated. The logic is clear here as well. Without this restriction, it would be possible to insert "malicious" libraries (ones that changed behavior of existing models) because they could trick the lookup process. For example, inserting a package called "Modelon.Modelica" would result in all libraries in the Modelon package having an alternative version of Modelica visible for all non-imported names. However, our original proposal included this, but my recent presentation and discussion on the topic left it out. For Modelica 3, this should be included. In Modelica 4, I think we should look seriously at getting rid of the "search up the hierarchy" part of the lookup rules.

If anybody has any other discussion points, please raise them. Otherwise, I'll be resubmitting something that is basically the same as what we put together at the end of the last meeting.

modelica-trac-importer commented 6 years ago

Comment by hansolsson on 28 Feb 2012 17:09 UTC From language group minutes (hopefully added to right ticket this time): Hans manually looked at Mechanics.Rotational and found at least the following:

Having Examples depending on several other packages is good for demonstrating the power of the library; the dependencies on Translational and Thermal interfaces is due to multi-domain models. However, some dependencies look more odd at first glance – they are due to the handling of optional mountings and similarly for the Thermal library; and as far as I can see they are in fact quite normal.

Obviously the examples in the proposal need to be revised based on this (having Examples as a separate group seems obvious); it might also imply more changes since a hidden assumption in the proposal was that the dependencies were simpler than this.

We need to look at other packages as well.

Conclusion: We should investigate other packages as well – and consider if there are other ways of separating models that are not directly tied to the package-structure.

modelica-trac-importer commented 6 years ago

Modified by dietmarw on 7 Mar 2012 14:12 UTC

modelica-trac-importer commented 6 years ago

Comment by dietmarw on 15 Aug 2012 08:22 UTC Left-over tickets from the now closed [milestone:ModelicaSpec3.3] milestone which basically means it should go into [milestone:ModelicaSpec3.4] right?

modelica-trac-importer commented 6 years ago

Comment by leo.gall on 23 Aug 2012 15:32 UTC See also Michael Tiller's recent [browser:"/ModelicaDesign/trunk/MeetingMinutesMaterial/min72_Linkoeping/Slides-and-Documents/Package Documents" documents] from Design Meeting 72 in Linköping.

Also, this topic influences the requirements of the proposed Library Distribution Protocol in ticket [ticket:536].

modelica-trac-importer commented 6 years ago

Changelog removed by leo.gall on 23 Aug 2012 15:32 UTC

modelica-trac-importer commented 6 years ago

Comment by hansolsson on 3 Dec 2012 15:20 UTC I believe we should reconsider the need for this proposal.

Let us start by looking at the problems with packages currently:

  1. Loading them completely takes time; and there might be parts that do not work in the tool.
  2. Massive synchronization to release a good version.
  3. Dependencies on multiple versions of libraries; with the risk of incompatibilities. (And odd conversions.)

So what are the standard ways to deal with this situations in other fields? • For 1 “Loading large libraries”: the solution is incremental loading; simple just delay it until needed. • For 2 “Massive synchronization”(see * below): the problem is unavoidable complexity and the solution is stricter release processes. We can shift the burden from library developer->tool developer->site admin->user; but we cannot avoid the problem – and in my opinion we want to detect the problems as early as possible. • For 3 “Dependencies on multiple versions”:

So what can we improve for Modelica: • The incremental loading is possible (and even implemented) in some tools in current Modelica; and there is nothing in the standard preventing this. Can demo if needed. • Improving processes for releases: YES! • Support multiple versions to allow bundling of libraries with other libraries? Note: we really want the exact version; including build# and versionDate. • Include conversion information in Modelica (alternatively standarize files) for more fine-grained control. Can demo if needed.

The main benefit of the package proposal in this is the incremental loading; but it does not make it easier to implement - it just assumes it is present.

Supporting multiple versions in the same session seems quite messy; it might still be needed. The package proposal does not make this significantly easier to implement this; we might use it to reduce the disk space needed (by skipping unused sub-libraries).

The conversion information is currently stored at the top-level (destination) packages in Dymola; I don’t see how we can easily distribute it.

*: We should also compare this with version control software. Here the trend has gone from versions on individual files (cvs) to versions on repositories/trees (svn, git). Basically the reason is that the old fine-grained versioning has not worked in practice.

We can still work on the package proposal, but to me it seems more beneficial to start working on the other problems.

modelica-trac-importer commented 6 years ago

Comment by mtiller on 5 Dec 2012 15:58 UTC The previous comment says "the main benefit of the package proposal in this is incremental loading". However, my proposal says nothing about incremental loading. Furthermore, it says nothing about loading times.

My proposal is about making it easier for model developers to manage complex sets of packages by breaking away from the paradigm that model libraries that have some kind of taxonomical relationship (e.g. all created by the same organization) need to distributed as a monolithic package. This is explicitly stated in the "Requirements" section of the proposal. The previous comment dismisses the monolithic nature as "unavoidable complexity and the solution is stricter release processes.". I completely disagree. While library development is complex, there is quite a bit we can do to manage this. The lookup rules, for example, are explicitly called out as something that unnecessarily complicates library development. Top-level versioning is another. All of this is discussed in detail in the proposal and none of that is refuted by these comments.

Finally, the issue of "dependencies on multiple versions" is also addressed in this proposal. The previous comment provides an aside on this topic but doesn't provide concrete proposals for addressing the problem and doesn't really refute the value of the concrete proposal already pending.

In summary, I feel that the previous comment doesn't address the core issues that this proposal explicitly states and tries to address nor does it actually refute the value of the proposal being made. As such, I think the original proposal is still a valid and tremendously useful solution for the problems it claims to address.

modelica-trac-importer commented 6 years ago

Comment by hansolsson on 5 Dec 2012 17:42 UTC Replying to [comment:11 mtiller]:

The previous comment says "the main benefit of the package proposal in this is incremental loading". However, my proposal says nothing about incremental loading. Furthermore, it says nothing about loading times.

Note that the three points above was extracted from the slide with 5 items from Johan; (two of the items correspond to issues caused by loading the entire library) i.e. I was trying to revist the original rationale for the proposal.

modelica-trac-importer commented 6 years ago

Comment by mtiller on 5 Dec 2012 18:10 UTC The original presentation from Johan has a slide titled "Problem" which defines the problem as "How can efficient management of large interdependent set of components be supported". That is the problem I am addressing. It makes NO mention of loading models.

Furthermore, he has a slide titled "Need" which attempts to decompose the problem into specific requirements for a solution. This maps exactly to the "Requirements" section of my proposal.

I agree that "massive synchronization" does come from his proposal but it is a description of the disadvantage of the current system and is manifestation of the problem as stated in that presentation (and the problem I'm attempted to address). Your analysis dismissed this as "unavoidable" and indicated this was an issue with the library release process. I disagree. This is not an issue of discipline. My position is that the language could make the life of the library developer substantially easier and it is NOT a lack of discipline or process that causes this problem.

I further agree that "dependencies on all versions" is an issue as well. I feel that my proposal addresses this issue quite nicely by reducing the "brittleness" of the current system (which is the central issue).

In summary, I think my proposal accurately addresses the underlying problem (as articulated by Johan and experienced as well my myself) and addresses the needs identified. It seems to me there are only two potential objections to my proposal: either it doesn't accurately reflect the statement of the problem or it doesn't solve that problem. It isn't clear to me which of these positions you are arguing.

modelica-trac-importer commented 6 years ago

Comment by hansolsson on 7 Dec 2012 12:53 UTC If we look at MSL 3.2.1 the current problems are currently very severe; and this proposal does not on its own solve those problems. It could be that it adresses a real problem for other libraries - but not for MSL.

Instead of looking at the proposal I thought it a good idea to go back to the issues with the current handling of libraries. That's just common sense: there is a list of issues with the current system; and it makes sense to investigate how such issues are normally dealt with - and also if the new proposal fixes them. That applies even if the issue is not listed in the rationale.

The reason it does not help for MSL is all MSL-versions starting with 3. should be backward compatible for users; this means that when a change in part of MSL breaks another part of MSL it would (normally) also break user-models and thus is a blocking regression.

That such regressions still happen are actually due to lack of discipline/processes. For my normal work we cannot commit directly to trunk; and this late in the development stage for MSL 3.2.1 it would be highly unlikely that we could commit for MSL 3.2.1 without a proper reason. (Technically it is not called "commit" and "trunk"; but the same idea - and this kind of process is not unique for Dassault Systemes. However, it takes extra effort and requires that someone actually handles it - "processes" will always be circumvented.)

modelica-trac-importer commented 6 years ago

Comment by mtiller on 7 Dec 2012 13:16 UTC To be clear, I agree that there are severe problems in the MSL. Those problems are related to the complex dependency relationships that have developed over time within the MSL. I also agree that some degree of discipline would help address the issues in the MSL.

However, the ticket is not about the MSL. The MSL was merely used as an example. Companies like Modelon and Ford have significant issues with trying to organize their models because the current semantics make it very difficult to be aware of these complex dependencies and, more importantly, do something to help manage it.

This is a proposal for improvements that help library developers manage their models. As such, I would appreciate if some of the library developers could read the proposal, as questions and then let us know whether they think this will be useful to them.

modelica-trac-importer commented 6 years ago

Comment by hansolsson on 7 Dec 2012 14:00 UTC Replying to [comment:15 mtiller]:

To be clear, I agree that there are severe problems in the MSL. Those problems are related to the complex dependency relationships that have developed over time within the MSL. I also agree that some degree of discipline would help address the issues in the MSL.

Can you provide examples of such problems?

When I think of the problems with MSL at the moment I see: unclear plans (3.2.1 was planned to be ready 2012-08-.. - it has changed, but I don't know to when, even if on map-lib), zombie trac-tickets (code corrected, but ticket not updated), sub-library being added late in release process and then removed even later (due to not following guidelines), min/max-values in SIunits causing regressions at least for user-models (possibly in MSL as well).

modelica-trac-importer commented 6 years ago

Comment by mtiller on 7 Dec 2012 14:49 UTC The main one would be the fact that there are, when you really analyze the libraries, a bunch of surprising and unanticipated dependencies. These have arisen because all the libraries simply depend on "Modelica XYZ" without any finer way to represent or restrict dependencies.

modelica-trac-importer commented 6 years ago

Comment by mtiller on 7 Dec 2012 14:55 UTC See the previous comment for a link to a revised version of the proposal in the new (proposed) MCP format.

modelica-trac-importer commented 6 years ago

Comment by hansolsson on 7 Dec 2012 17:02 UTC Replying to [comment:17 mtiller]:

The main one would be the fact that there are, when you really analyze the libraries, a bunch of surprising and unanticipated dependencies. These have arisen because all the libraries simply depend on "Modelica XYZ" without any finer way to represent or restrict dependencies.

I would say they are minor issues compared to lacking an updated release-plan for MSL 3.2.1, or having an MSL 3.2.1 trunk that breaks other libraries.

An example of surprising dependencies was added 9 months ago in this ticket (the first sub-library I looked at):

Mechanics.Rotational.Interfaces depends on SIunits, Thermal.Interfaces, Mechanics.Rotational, Mechanics.Translational and Mechanics.Translational.Interfaces

The ones in italic are due to the optional mountings.

modelica-trac-importer commented 6 years ago

Modified by dietmarw on 13 Oct 2014 09:55 UTC

modelica-trac-importer commented 6 years ago

Comment by dietmarw on 16 Nov 2014 00:28 UTC In preparation of the milestone:Design84 I've created a test library which makes it easier to visualise the dependencies including circular ones of the different sub-packages of the MSL: https://github.com/dietmarw/M_S_L that repository also contains some structure*.graphml files that can be opened with yEd to analyse the dependencies easier.

As a note depending on the outcome of the discussion of this ticket and the way it is resolved we need to revisit tickets #89, #132, #1023 and finally #1491. In case I forgot some please add a comment with a link to them.

modelica-trac-importer commented 6 years ago

Comment by sjoelund.se on 4 Dec 2014 13:41 UTC I chatted with Dietmar a bit regarding the issue of Resources/Library. Since each sub-package is independent of the others, there needs to be a good design pattern for handling resources shared by several sub-packages.

A good solution seems to be to add a sub-package only for resources, in MSL: Modelica.Resources

So external functions use:

Library="ModelicaExternalC", LibraryDirectory="modelica://Modelica.Resources/Library/"

This is needed because the default is modelica://LibraryName/Resources/Library, and since the meta-package Modelica might not be loaded (or even existing), modelica://Modelica/Resources/Library would not resolve.

The design pattern seems to solve all major issues, and does document the dependency in a good way (most MSL packages would have a uses annotation for Modelica.Resources).

Note: Most tools seem to add ModelicaExternalC somewhere in the compiler library path rather than in the resources directory, so the LibraryDirectory might not be strictly necessary in order to work on most tools. But it is necessary if the tools do not do this (so it should be there).

modelica-trac-importer commented 6 years ago

Comment by dietmarw on 19 Dec 2014 07:01 UTC Ticket retargeted after milestone closed

modelica-trac-importer commented 6 years ago

Modified by otter on 23 Jan 2015 09:29 UTC

modelica-trac-importer commented 6 years ago

Comment by dietmarw on 15 Apr 2015 14:33 UTC Ticket retargeted after milestone closed

modelica-trac-importer commented 6 years ago

Modified by dietmarw on 20 May 2015 08:13 UTC

modelica-trac-importer commented 6 years ago

Comment by leo.gall on 24 Apr 2018 11:53 UTC This ticket is quite important, very long, but seems to be dead? I would like to link to the other discussion:

1728 new enhancement MCP-0016: Semantic Versions for Libraries

modelica-trac-importer commented 6 years ago

Comment by hansolsson on 24 Apr 2018 12:44 UTC Replying to [comment:27 leo.gall]:

This ticket is quite important, very long, but seems to be dead?

I guess a major reason is that few are interested in working through the details (and it is not the only such ticket). It could also be that some changes in how tools handle version have reduced the requests.

If the sub-libraries of MSL were separate it would be easy to split as proposed in this proposal, but most are not. Additionally it is not clear how many of the sub-libraries are constant between releases.

Does that mean that the proposal is way too fine-grained (so that we should instead have a handful of sub-libraries in total), the libraries should be restructured in another way, or something else?

The user benefits are also less clear. We currently release e.g. MSL 3.2.2 as a synchronized release and expect tools to use that - which makes it easy for users to know what they got - and commercial suites are often also similar. Is the plan to move away from a synchronized release to something else? If so, how is that supposed to work?