Tracking issue migrating to non-proprietary UML-based workflows

[SergejMuhic]

I would like a satisfying answer on this topic. My impression from the Technical Road map document was that the IFC Specification (maintenance) is currently too bespoke and we need something that is more open to maintain the spec. I have tried opening the EA XMI format in various UML tools such as Altova, Modelio etc. and failed if not deleting the Enterprise Architect extension node.

So, in order to resolve the "bespokeness" the move to a not only bespoke but also proprietary format (which is a kind of a paradox in itself on so many levels) is questionable. I am not even sure the bespoke nature is problematic because IFC is a bespoke standard. In order to document it, it cannot be avoided as is the case in XMI now.

This is why I was asking for a standard requirement analysis to see what the goals are and a regression matrix to compare the features of the systems. Which ones will be kept which deprecated and what the replacement will be or a statement why it is not needed anymore. I can understand the push to EA if there is a clear interest to maintain the schema as an EA model but if that was not the objective and we resolve nothing and getting a system that is inferior in critical features that were mentioned in the emails and our meeting on Nov 4th, I do not understand this push.

For the sake of transparency, I would propose to have a requirement analysis where the objectives are listed and we have a way to measure that these objectives were accomplished and a regression matrix with an explanation why the features that were mentioned in the emails and the meeting on Thursday Nov 4th will either not be supported, or if they will, when. Otherwise, there is no accountability and no means of planing in the projects as well as implementation of bSI standards.

[aothms]

This is a duplicate of https://github.com/buildingSMART/NextGen-IFC/issues/2 which unfortunately wasn't migrated to this repo.

It's quite a stretch to go from "deleting the Enterprise Architect extension node" to "proprietary format".

We rely on tags and stereotypes for some of the IFC-specific conventions. These are in a custom profile, not solely in the ea extensions. Tags and stereotypes are a core part of UML/XMI.

I'll put in some additional effort to demonstrate how to interoperate with papyrus. It may need some light xslt'ing.

[SergejMuhic]

This is not what I understand under a satisfying answer. The point was not on the format, I just made it as an example to connect between the Technical Road map and the actual implementation. We can discuss many more (such as CI/CD that is not supported with this XMI format) but that is beside the point of this inquiry.

My point is to get transparency which would allow planing in projects and implementation of the standard. What are the features that will be available comparing the systems. Which ones when, the ones that will not be, why and what is the replacement. My proposal for that is to use the bSI process: compile a requirement analysis with a regression matrix, make a project plan ...

Otherwise, there is no prioritization, no accountability, no transparency in decision making and no way to plan anything that should be based on the IFC as a standard.

[berlotti]

The move to represent IFC as a UML class diagram is a given. It is a requirement that has been made by many stakeholders (governments, partner standardization organizations, academia, technical room working groups, etc..) for many years. It is impossible to have a perfect solution from one day to the next. The current situation is a step based progression to a better situation than before. The developing nature of the standard through projects and community efforts makes it difficult to plan exact functionalities both in the standard as in the maintenance system. There has been a dependence on project that used legacy systems. With that out of the way the first priority is quality control (see the scripts) and generation of the ISO delivery package (http://ifc43-docs.standards.buildingsmart.org/).

[SergejMuhic]

The move to represent IFC as a UML class diagram is a given.

Sure. All good, can be done in many tools and does not mean that the spec shall be stored in XMI. XMI as an exchange format can be generated from the spec and imported back. This would produce an actually exchangeable format that could be opened in many tools not that we rely on one specific tool (EA). XMI not allowing a CI/CD process would be a blocker in my view. It is the same issue as using .ifc to archive data. Not really the best option. The development of the standard is not scalable and a bottleneck is being created.

It is a requirement that has been made by many stakeholders (governments, partner standardization organizations, academia, technical room working groups, etc..)

In that case a requirement analysis should be straight forward and a project easy to setup with this many stakeholders. This way prioritization of features could be done.

It is impossible to have a perfect solution from one day to the next.

Agree. Let us work toward it.

The developing nature of the standard through projects and community efforts makes it difficult to plan exact functionalities both in the standard as in the maintenance system.

Understood. In my view though, some predictability is better than none.

quality control

Why is this prioritized? If it is prioritized, where can I help out to make it uncover real issues. Currently, the schema comparison seems to not be working correctly or I am confused on the output. In any case, this should be clarified. The issues posted automatically seem to favour no documentation for attributes. Since when is it a requirement to have documentation for every attribute? E.g. PredefinedType is kind of repeating itself.

ISO delivery package

This is not comforting as it seems not compliant with the ISO spec e.g. contributors, captioning etc. There are also issues with that representation, some documentation and images are missing. The inheritance of usages is also off. Can we have a session on this and go through the problems?

[billeast]

How is it that bSI will provide copies of EA and the repository to project teams? How will comments across multiple projects be visible to all? Given that projects are scraping by with funds, this does not seem like a prudent business decision by bSI to pay a 3rd party software company when -OPEN SOURCE- solutions have been successfully used for many years. Has Richard P agreed to add this overhead cost to all future projects? I don't suspect that bSI management is even aware of the tax being imposed for a benefit that has yet to be realized.

[berlotti]

There are a lot of assumptions in your question Bill. The facts: EA is providing free licenses to bSI (that formally approved projects will be able to use). Using open source tools like papyrus will also become an option.

[billeast]

Thanks for the update Leon! Look forward to seeing what new capability that the new platform has and training about how teams can directly use it instead of the existing platforms. What is the plan to migrate existing working repositories to this new platform? What is the timeframe? I hope it will be available for FMH-EM!

[anborr]

My concerns with the new workflow are:

1. The UML/EA modeling guidelines of bSI are not publicly available. The well-documented and established modeling guidelines of the Infra and Railway Room seem to be obsolete.

2. EA cloud repository is not used, but instead there seems to be one EA master copy on one particular machine. This will prevent any kind of concurrent development in projcts running in parallel. Instead, we will have to perfom manual merging and remodeling.

3. Having XMI as one large file as the main item on the repository prevents fine-grained version control. The functionalities of Git for supporting concurent development cannot be used.

4. I agree with @billeast that dependecy on commercial systems such as EA is a cost risk for bSI and/or the projects.

[berlotti]

1. they are documented as python quality control scripts. PDF will follow asap.
2. there is no EA master copy. There is no EA file. The XMI is all there is. As described earlier in this thread it is the intend to keep it interoperable with other UML tools.
3. current version control seems to work pretty fine grained. Version updates from Github work fine due to the agile nature of the agnostic XMI.
4. there is no dependency on any system; not commercial and not non-commercial. This UML setup also has no dependency on only one poorly maintained tool. That would go against everything buildingsmart stands for.

[anborr]

1. I would see the PDF guidelines as a prerequisite for turning the new workflow into production. In the moment I would still see it as being experimental.

2. Good to know! But then the question appears: Are the developers expected to re-create the class diagrams from the model? The provided XMI is inconsistent in this regard: It contains 22 (more or less random) diagrams, which are by no means complete. See lines 664898ff as an example: That's the list of diagrams:

3. It's a file with 665.068 lines! Any tiny change will results in the entire file being marked as modified. In usual Git workflows modifications are kept local to individual files, allowing smooth merges across different branches (which is particular import for concurrent development in different projects). Even more importantly, other tools than EA may (and will) have different serialization order. Exporting from these tools and commiting into the repo will result in a massive number of changes which however are trivial, but at the same time cannot be identifed neither by diff tools nor by humans. I expect that we will see 600.000 lines being affected.

4. Line 3 says: <xmi:Documentation exporter="Enterprise Architect" exporterVersion="6.5" exporterID="1559"/> For me it seems that the XMI is full of EA specifica which are hard to be reproduced by other tools.

5. 5547 lines have author='tkrij'. Others have "IQSOFT" or "IFC Rail Team". Is this supposed to be part of the standard?

This is not fundamental critique, but serious concerns. My strong impression is that the workflow requires more refinement, before it can become mandatory for all bSI projects.

[aothms]

Line 3 says:

5547 lines have author='tkrij'. Others have "IQSOFT" or "IFC Rail Team". Is this supposed to be part of the standard?

22 (more or less random) diagrams

The exporter related attributes are just meta-data as to which tool produced the model. That's not a concern. The project and authorship (author attribute) information is part of the EA extension and can be disregarded, it's not considered in the artefact production processes. Similarly other tools will ignore it. Same applies to the diagrams. Part of the extension. It's there because there are some longer running topics, such as tesselated faceset textures, that needed several iterations so we kept the diagrams temporarily intact, but will be removed again from the output. There is no dependency on it.

massive number of changes which however are trivial, but at the same time cannot be identifed neither by diff tools nor by humans

This is a valid concern but has been discussed several times. For the submission to ISO we just haven't prioritised it.

The model will be exported more granularly, probably by package. EA extensions can be stripped out once the stereotypes made it to a uml:Profile, which would literally decimate the XMI file size. We will likely have some sort of git commit hook to canonicalize the xml (very light massaging: apply deterministic identifiers, sorting nodes and attributes if needed). But simple fact is that current priority are the 140 issues open still in this repository that need adjustment in the specification in a relatively short time frame.

PDF guidelines

The same applies here. Once the tasks above are implemented we will obviously publish a guide.

[berlotti]

Projects are not mandated to use this since projects are not releasing IFC.

This issue is full of assumptions and misconceptions. There seems a resistance to things that are not understood. That is not an argument to slow things down however.

[SergejMuhic]

This issue is full of assumptions and misconceptions.

I feel like I have to clarify the original post. It feels bad that I need to defend myself for just stating my concerns looking at what is publicly available instead of getting clear answers. If anything stated is wrong, please explain why and how it is supposed to be resolved instead of talking down on the issue. Shooting the messenger has historically not helped.

There seems a resistance to things that are not understood.

Asking valid questions and stating concerns is not resistance. I have stated many times that I have no feud with UML or XMI (or any other technology or tool). My concern lies with how they are used, the process behind and communication. In that I am not alone. Assuming that I do not understand certain things is just what it is. That is why questions have been asked. For me, inexact replies do not not build confidence.

On to the points:

1. they are documented as python quality control scripts. PDF will follow asap.

If python scripts are "the norm" I would appreciate some documentation in them. As it currently stands, I can only find some "todo" and "notes to self" such as: https://github.com/buildingSMART/IFC4.3.x-development/blob/a142882f607b09dc8cda4e1e69420369e0657cff/code/xmi.py#L136

2. there is no EA master copy. There is no EA file. The XMI is all there is. As described earlier in this thread it is the intend to keep it interoperable with other UML tools.

As @anborr states, the XMI from 897xx to 6180xx (85%) is the "Enterprise Architect" metadata that is not documented (link points to "Page cannot be found") and is probably the reason why (as I said above) it is not compatible with UML tools other than EA.

3. current version control seems to work pretty fine grained. Version updates from Github work fine due to the agile nature of the agnostic XMI.

This 0b96fb1b2e74db1b89c1e27db5c01c405148cf87 is not fine grained. '8,160 additions and 8,151 deletions' for an added inverse.

Regarding agnostic, and as was stated above, the scripts are the documentation, I would point to: https://github.com/buildingSMART/IFC4.3.x-development/blob/63e1aeb1ad20a9a6df772ac27c6b0cc099b1ee17/code/to_express_repo.py#L176 and

<tag xmi:id="EAID_CC98D225_8180_1B19_E053_5801400A16D7" name="ExpressOrdering" value="0003"/>
<tag xmi:id="EAID_CC98D225_8182_1B19_E053_5801400A16D7" name="ExpressOptional" value="OPTIONAL”/>

as one example. Just clarifying why I could have been under the impression that the EA extension is used.

4. there is no dependency on any system; not commercial and not non-commercial. This UML setup also has no dependency on only one poorly maintained tool. That would go against everything buildingsmart stands for.

Currently, the dependency is definitely there (see references above). If you have plans on changing that, sure, I would welcome that, but it does not reflect the state-of-the-art. Again, I can only base my concerns on that.

I welcome that bSI finally realized that the maintenance of their tool is poor. I hope this also means stepping up to their responsibility. In the meantime, the "poorly maintained tool" is what we have.

Take this post as a clarifying the facts of the matter. It is not critique of any sorts, just a reflection of what this repo exposes to me. Any wrong statements on my side are a consequence of that. Not published scripts, documentation, decisions, solutions etc. are not known to me and I may stand corrected.

@aothms thank you for sharing some thoughts. These are, however, not official and hence hard to incorporate into daily business of any project which is part of what I was trying to address originally.

[anborr]

This issue is full of assumptions and misconceptions.

In an open community it should always be possible to ask questions and raise concerns. The main reason for these concerns may lie in the fact that new workflow and its underlying concepts and assumptions are not publicly documented. Given the importance of the workflow, an official bSI document would be very welcome. Other than that, the management of 600.000 lines file could be easily interpreted as a misconception.

Projects are not mandated to use this since projects are not releasing IFC.

That’s correct, but projects are developing extensions and/or modifications. In most of the Git projects in the world, supporting a fine-grained development process is the exact reasons for using Git, as it allows to formulate and discuss pull requests on the level of individual modifications. Normally, Releases are just milestones in the development process – a well-supported Git concept. If I understand this comment correctly, the development process itself is not supposed to take place in this repository. At the same time, according to instructions by the bSI Technical Director, bSI projects are indeed demanded to deliver their results into this repository. For the reasons indicated above (one large XMI) this would not be possible on the level of individual changes, but rather as a bulk upload, where changes are not traceable. Is this the intended workflow?

[anborr]

4. There is no dependency on any system; not commercial and not non-commercial.

I attach the generated output of Visual Paradigm, created by importing the XMI and exporting it again without applying any model changes: VPP-IFC-XMI.zip According to the comments above “no dependency on EA”, "no master file", “just meta-data”, I would assume that this would be a valid commit into the repository? DiffMerge marks all lines as being modified.

If not, what are the exact requriements for the XMI to be accepted as a project deliverable?

[anborr]

In summary, I feel that at this time (Feb 2022), there are too many open questions/issues to use the workflow for bSI projects in development mode. This not an argument against the concept per se nor for slowing down, but in contrary an apel to speed up to make the workflow mature enough that it can be used in real concurrent IFC development. Unquestionable requirements are the exact definition of the XMI to be delivered and a much more fine-grained file structure to allow concurrent development. I understand that other issues may have higher priority - but then we should be consequent in stating that the process is still experimental and not demand delivery into it from bSI projects.

[berlotti]

Project proposals don't go into UML directly, but follow a defined process. Passing room steering committee, SCTE, SCE, etc. Domain parts and parts on the recources and interoperability layers are treated differently. Interoperability parts and recourse layer are not under the mandate of the project but the SCTE. Most projects are being payed by buildingSMART and will therefore get instructions.

Available answers have been given in this issue. Sorry if you don't like them. Some answers are not there yet. Sorry for that as well. Anyone is welcome to help work on them.

[Moult]

BTW looking at the original thread https://github.com/buildingSMART/NextGen-IFC/issues/2 I saw that @hlg had a go at stripping it down and I dug back in time and found https://github.com/buildingSMART/IFC4.3.x-development/blob/a6ab82adf6a3c048e732c9df71e801c022f19713/schemas/HarmonisedUMLmodel.xml - however it seems to be 42MB still. Did I find the right one? At a glance in an editor I don't see much difference in terms of namespaces and tags, but I didn't dig too deeply.

I downloaded Modelio and was unable to figure out how to load the XMI in. I also got Umbrello loaded and I was able to open up the XMI. I didn't get much further but I know nothing about authoring UML so that's probably the reason :)

Just to throw in some positivity I'm very excited in this being a step in the right direction. IfcDoc as far as I was concerned relied on a proprietary OS so IfcDoc was pretty unusable (yes, C# but I know). Moving to a tool-agnostic (in theory, obviously not there yet) data model is also super cool. Separating written docs from schemas as Markdown? Huge win. Version control (right now only practical for written docs, not schema, it seems) also a big modernisation step. The big moves are all the right ones.

@aothms says: The model will be exported more granularly, probably by package. EA extensions can be stripped out once the stereotypes made it to a uml:Profile, which would literally decimate the XMI file size. We will likely have some sort of git commit hook to canonicalize the xml (very light massaging: apply deterministic identifiers, sorting nodes and attributes if needed). But simple fact is that current priority are the 140 issues open still in this repository that need adjustment in the specification in a relatively short time frame.

Now 168 open issues :) But I'm curious, would this be just 1 smaller XMI that would still need conflict resolution every time there is a merge, or would it be many separate but linked XMIs that people can work with independently in different tools? (That sounds more pleasant to me)

I know this thread is about the UML but because the schema is only half the story and the docs is the other half I've started poking around the code and updated this readme file I'd like to point people to - it's an absolute joy to be able to edit and run the documentation live on a fully non-proprietary stack.