Are multiple dependencies on different versions of the same library allowed in Modelica?

[casella]

Consider the following scenario

• package A uses packages B and C
• package B uses package D version x
• package C uses package D version y
• package D version x has no conversion(noneFromVersion="y")
• package D version y has no conversion(noneFromVersion="x")

In principle, a tool could load both versions x and y of D, and have packages B and C use the corresponding ones while flattening classes in A. AFAIK, there is no explicit provision in the Modelica Specification barring this option.

In practice

• This could quickly become quite messy for users and developers to figure out
• No Modelica GUI I know of is capable of loading multiple versions of the same library at the same time

As @sjoelund argues here, the fact that the capability of handling multiple versions of the same library is not explicitly mentioned means that a library developer should avoid relying on it, to ensure maximum compatibility.

However, I believe it would be better to explicitly bar this option in the language spec, because this could be a serious hurdle to ensure cross-tool compatibility of future Modelica libraries, which is the whole point of having an open standard.

For example, we could add to the normative part in the preamble of Section 13.3 the following statement:

When flattening a given class, only one version of any given library A in the MODELICAPATH can be considered during the entire flattening process.

We can leave it to the tools how to figure out what is the best or most appropriate version they should consider, but at least we should make clear it has to be only one.

@HansOlsson, @sjoelund, @mtiller, @dietmarw, what do you think?

[eshmoylova]

I would say that it is covered by 4.2 Double Declaration Not Allowed. If you load two versions of the same library you would have two packages with the same name in the global scope. But I can see how it can be argued that they are not declared but loaded, so the rule may not technically apply.

[svorkoetter]

I'm with the first option: it's not allowed.

[sjoelund]

I'm thinking that having this would make conversion script less mandatory. Today, when you convert a library, you need to create a new conversion script and anything using your library also needs to be converted, creating a huge chain of conversions. Some of these libraries might not be maintained anymore and you then get the problem that you are referring to some converted version of the library that no-one else has access to.

By allowing multiple versions of a library to be loaded, you could simply keep on using libraries without updating them. But it would be quite confusing to a user of a GUI to have multiple MSL versions open and depending on which diagram you are in, only one version would be possible to use.

[henrikt-ma]

I like @casella's original idea of targeting rules for flattening a given class, rather than entering the domain of how tools should manage available versions of libraries between model translations.

[casella]

By allowing multiple versions of a library to be loaded, you could simply keep on using libraries without updating them. But it would be quite confusing to a user of a GUI to have multiple MSL versions open and depending on which diagram you are in, only one version would be possible to use.

This is true in principle. In practice, using stale libraries that create multiple-version dependencies is something I would not leave to a fully automated systems, but rather to experts, which may be able to untangle the dependencies.

I'm also in favour of allowing only one version. BTW, I guess that is what other languages which heavily rely on package management like Python or Julia do, isn't it?

[sjoelund]

With Python in particular you are recommended to have one virtualenv with all the packages you need for one particular library in one location since the libraries typically break a lot on update... It's not a very good example and it's sometimes difficult to upgrade a certain package if it has the same dependencies as something else that your code depends on. So you are often stuck with 1-2 year old (or older) libraries since Python only works if all libraries are actively maintained and updated to the latest dependencies...

If we do the same, we will be stuck with requiring our models to use old MSL and not able to upgrade because a library we depend on isn't upgraded to the newer MSL versions.

[HansOlsson]

TL;DR: Having multiple versions together should be forbidden. I would say that it is already forbidden (but we can make it more explicit) as the used libraries in MODELICAPATH are loaded in the global scope, and the global scope can only contain one version of each library, https://specification.modelica.org/maint/3.5/scoping-name-lookup-and-flattening.html#static-name-lookup

Longer: As I understand having multiple versions of libraries would be somewhat easier if they are only used internally in other libraries, and not directly exposed to the user. I've heard that Lua has that (using internal tables), but I cannot find any link at the moment.

However, since we want libraries to expose Interfaces for interoperability I don't see that internal use inside libraries will be overly useful for Modelica. Thus if we want handle multiple versions of a library we also need to handle:

• Multiple versions of a library used in the same model. I can see some possible syntax - but we don't have it yet.
• Leading to user confusions when you have different versions of the same model; especially as they look similar and might even behave identically and/or have the same contents. (Yes, the models may have identical texts, but depend on other classes that differ in contents and/or behavior.)
• Especially as many operations (Create Connector, Duplicate, Propagate, ...) copy the component-class references, meaning that this confusion will propagate. It is not even certain we can connect the different versions together.
• Explosion in number of versions. We are here discussing version x and y; but when can have multiple versions we might end up with: x, y, z, w, a, b, x.y, etc, and so on.

If there are any issues with converting libraries I think we should work on that instead.

BTW: Dymola can support the similar scenarios as the one in original post (except that libraries should be ordered), but the classes using outdated versions will be read-only (as I recall we can do it even if there are conversions in the case they wouldn't impact your library). The read-only part is needed because even if no conversion is needed to convert to the new version it might have introduced a new class or parameter, that didn't exist in the older version and keeping track of that would be too complicated.

[sjoelund]

However, since we want libraries to expose Interfaces for interoperability I don't see that internal use inside libraries will be overly useful for Modelica.

There is no nominal typing except for external objects, right? The interfaces should be compatible since they just boil down to the type and unit.

The most problematic part would be modifications on components (like redeclaring Media).

If there are any issues with converting libraries I think we should work on that instead.

The problem isn't converting the libraries as much as it is making people convert their libraries...

[HansOlsson]

However, since we want libraries to expose Interfaces for interoperability I don't see that internal use inside libraries will be overly useful for Modelica.

There is no nominal typing except for external objects, right?

There's also for operator overloading. However, my point was slightly different:

The interfaces should be compatible since they just boil down to the type and unit.

In many cases, but I could see that a new version renamed some connector-variables; and/or changed their type.

The problem isn't converting the libraries as much as it is making people convert their libraries...

Well, similar issue. To me one of main issues is that people are hesitant to do it - and also hesitant to use the new possibilities to simplify their work-flow.

[sjoelund]

I suppose one thing we could specify and recommend would be specifying how to handle older libraries where a conversion script is available for these older libraries. Basically if we have uses OldLibrary which uses MSL 3.2.3 and a model with

uses(Modelica(version="4.0.0"), OldLibrary(version="1.0.0", conversions(Modelica(version="4.0.0")))

This would make it easier to use newer libraries without having write access to their repositories.

[casella]

Based on the discussion so far, could we agree on the following scenario:

• We make it more explicit that when flattening a model, only one version of each library can be considered, for clarity
• This means GUIs must only display one version of each library (as they do today) and we avoid nightmares to people that would otherwise find this very messy
• Tools should try to figure out how and were to get versions that are mutually compatibile with each other without conversions and are as up-to-date as possible, leveraging as much as possible on the noneFromVersion annotation - how they do this is a tool issue, not specified
• In case this doesn't work because of old libraries with incompatible dependencies (e.g. MSL 3.2.3), people could add to their libraries using them an annotation with the required automatic conversion, that will be performed on-the-fly when the library is loaded, without asking for confirimation. This may not work in all cases (sometimes some minor manual fixes are unfortunately required) but would be much better than just throwing the old library away.

I think this is the most reasonable compromise in terms of being comprehensible to end users, while being reasonably easy to implement for tool vendors.

Do you agree?

[henrikt-ma]

It would be easier to answer if the question was clarified with a clear separation of what is intended as:

• Normative specification content.
• Non-normative content.
• Rationale for the sake of this discussion that doesn't go into the specification at all.

My gut feeling is that we (the specification) should not elaborate too much on this beyond what is normative.

[HansOlsson]

Language group: Already restricted (to only one version of a library at the time) - but make it clearer. @eshmoylova where?

[eshmoylova]

I am afraid that my looking into where to add the restriction raised more questions, as it often happens. The following are the places that I think need to be updated (but may need some discussion).

1. In Section 18.8.2, it says that one or more uses annotations are allowed. But it does not say if more than one uses annotation can specify the same library with different versions. I think it should be specified if it is allowed and, if it is, what it means:

   • Compatible with both version. Use the latest one you have available. I guess it would be equivalent to conversion(noneFromVersion = "earlier version"). But having an option to write it using the conversion annotation does not preclude anyone from writing two uses annotations anyway. So it's good to have a rule on how to deal with it.
   • Keep only the latest (in order of how it is written) and disregard any other ones.

2. "Global scope" is used in many places in the specification, but I cannot find a clear definition of what constitutes the global scope. In Section 13.3, it says (the emphasis is mine): "The top-level scope implicitly contains a number of classes stored externally. If a top-level name is not found at global scope...." I think we need to define "global scope" in Chapter 5. After Enclosing Classes, we can have a section Global Scope, saying, for example, that the global scope contains built-in operators (defined in the specification and tool internal), loaded libraries using MODELICAPATH (link to 13.3), and loaded models and classes via File Open or other tool-specific mechanism. Then either here or in 13.3 we need to say explicitly that only one version of a library is loaded into the global scope per simulation or translation. @casella's suggestions was "When flattening a given class...." But that might imply that we can/should rebuild the global scope when we flatten a new class. For example,

   model M
   A a;
   B b;
   equation
   b = someFunction(a);
   ...
   end M;

   When flattening M we need to flatten A and B. We want to have just one scope that applies to all those flattenings. Also, evaluation of equations happens after flattening. When we evaluate equations we need to flatten someFunction. Technically, it can be viewed as flattening a different class. So only limiting loading of only one version of a library, when flattening a given class, leaves a way out of the restriction.

3. Also we have The Class Tree and it should be tied to the global scope. Right now it says, for example, "The builtin classes are put into the unnamed root of the class tree." The unnamed root of the class tree represent the global scope. So maybe it should say so explicitly. Also where would the Instance Tree go? Does it matter?

[HansOlsson]

@eshmoylova

1. I would say a specific library should only appear once in a uses-clause. As I see it allowing a dependency on multiple versions is both difficult to maintain when changing, and puts too much emphasis on compatibility so there's a risk that developers go for the common ground.
2. Yes, I think the restriction "when flatting a given class..." makes sense, since that allows multiple versions on the file system.
3. Yes, we should make sure that the global scope is the top of the class tree.