modelica-tools / FMUComplianceChecker

FMI Compliance Checker for validation of FMUs 1.0 and 2.0
BSD 3-Clause "New" or "Revised" License
43 stars 31 forks source link

What is the scope of the Compliance Checker? #16

Open modelica-trac-importer opened 6 years ago

modelica-trac-importer commented 6 years ago

Reported by andreas.junghanns on 12 Nov 2015 12:13 UTC It is currently not clear to everyone what the Compliance Checker should be used for.


Migrated-From: https://trac.fmi-standard.org/ticket/355

modelica-trac-importer commented 6 years ago

Comment by andreas.junghanns on 12 Nov 2015 12:25 UTC Mission for the CC: Enhance compliance to the standard and improve compatibility of import and export tools.

Is the Compliance Checker able to validate FMI compliance? Currently: No. That means passing the Compliance Checker does not mean an FMU is *fully* compliant!

As long as we are not at a fully compliant check, we need to list what is currently NOT checked.

Proposal: All improvements to the CC are striving to achieve a fully validating verdict.

The way there will require a lot of work and the list of unchecked properties is a guide for the road.

modelica-trac-importer commented 6 years ago

Comment by karl.wernersson on 13 Nov 2015 13:08 UTC I will start list for a full check (without any priority order), some might already be checked so please correct me if this is done. The Idea is not that all of this needs to be done now some things are maybe not so crucial, but a list can help to assist in documentation of what the compliance checker tests and what it doesn't test

XML parsing

FMI 2.0

FMI 1.0

Functionality Testing

FMI 2.0 Most of the basic functionality it tested by a working simulation but some functionality are maybe not tested in a basic case and listed her.

Testing of optional Features (if suported)

modelica-trac-importer commented 6 years ago

Comment by edo.drenth on 21 Apr 2017 16:16 UTC Let me start with saying that testing is very important. But, I am unsure if it is possible to have one (1) compliance checker. The experience I have built up over the past years tells me differently. I run received FMU's in three different tools; always. The tools are all based on the same specification, but relatively frequent, the interpretation is different or pure programming bugs are revealed. Now that we have accumulated mileage in the tools, wouldn't it be better to have a mandatory pool of tools that you would need to test your produced FMU in for checking compliance? The actual tests can be based on the test suite currently in the CC.