modelica / ModelicaStandardLibrary

Free (standard conforming) library to model mechanical (1D/3D), electrical (analog, digital, machines), magnetic, thermal, fluid, control systems and hierarchical state machines. Also numerical functions and functions for strings, files and streams are included.
https://doc.modelica.org
BSD 3-Clause "New" or "Revised" License
478 stars 169 forks source link

Regression testing for MSL 3.2.1 #1114

Closed modelica-trac-importer closed 7 years ago

modelica-trac-importer commented 7 years ago

Reported by otter on 20 May 2013 18:32 UTC I run a regression test of MSL trunk (MSL 3.2.1) with respect to MSL 3.2 with Dymola 2014. The first list "Differences that indicate errors" (about 20 models) needs to be inspected by the respective library officer. The second list "False validation failed" is for your information (but is uncritical):

Differences that indicate errors that need to be investigated

False "Validation failed" (models seem to be o.k.).


Migrated-From: https://trac.modelica.org/Modelica/ticket/1114

modelica-trac-importer commented 7 years ago

Comment by jziske on 21 May 2013 07:43 UTC Modelica.Magnetic.FluxTubes.Examples.SolenoidActuator.ComparisonQuasiStationary Variable advancedFeed_x.v of MSL 3.2.1 is more accurate then of MSL 3.2 due to changes done for ticket #942. So no problem here.

modelica-trac-importer commented 7 years ago

Comment by otter on 21 May 2013 09:02 UTC Modelica.Blocks.Examples.BooleanNetwork1 fixed in 1698e11dd8df937c200957ca170829f378ac4445

modelica-trac-importer commented 7 years ago

Comment by ahaumer on 21 May 2013 12:49 UTC Modelica.Electrical.Machines.Examples.AsynchronousInductionMachines.AIMC_withLosses Different Initialization, correct behaviour

Modelica.Electrical.Machines.Examples.DCMachines.DCPM_withLosses Have to correct Parameterization

Modelica.Thermal.FluidHeatFlow.Examples.PumpAndValve Have to correct Parameterization

modelica-trac-importer commented 7 years ago

Comment by ahaumer on 21 May 2013 17:42 UTC partly resolved for Modelica.Electrical.Machines.Examples.DCMachines.DCPM_withLosses and Modelica.Thermal.FluidHeatFlow.Examples.PumpAndValve by 6f7f89350815bc8e1f60ab2bf47d806b894b508f, Modelica.Electrical.Machines.Examples.AsynchronousInductionMachines.AIMC_withLosses is due to new initialization.

modelica-trac-importer commented 7 years ago

Comment by msielemann on 24 May 2013 11:57 UTC Modelica.Fluid.Examples.BranchingDynamicPipes

modelica-trac-importer commented 7 years ago

Comment by msielemann on 24 May 2013 12:07 UTC Modelica.Fluid.Examples.HeatExchanger.HeatExchangerSimulation

modelica-trac-importer commented 7 years ago

Comment by msielemann on 24 May 2013 12:32 UTC Modelica.Fluid.Examples.TraceSubstances?.RoomCO2WithControls

modelica-trac-importer commented 7 years ago

Comment by msielemann on 24 May 2013 12:47 UTC Modelica.Fluid.Examples.PumpingSystem

modelica-trac-importer commented 7 years ago

Comment by msielemann on 24 May 2013 13:00 UTC Modelica.Fluid.Examples.HeatingSystem

modelica-trac-importer commented 7 years ago

Comment by msielemann on 24 May 2013 13:35 UTC Modelica.Fluid.Examples.Tanks.TanksWithOverflow

modelica-trac-importer commented 7 years ago

Comment by msielemann on 24 May 2013 14:00 UTC Modelica.Fluid.Examples.AST_BatchPlant.BatchPlant_StandardWater

modelica-trac-importer commented 7 years ago

Modified by msielemann on 24 May 2013 14:06 UTC

modelica-trac-importer commented 7 years ago

Comment by hubertus on 24 May 2013 14:20 UTC Water IF97 has an approximate numerica inverse that is start of the standard. 0.02 K difference is probably fully inside the accuracy spec of that inverse. It is fine to ignore that difference.

modelica-trac-importer commented 7 years ago

Comment by rfranke on 24 May 2013 15:34 UTC Replying to [comment:10 msielemann]:

\* Analyze rationale for change in [5946] and consider reverting this change

The rationale for the change in [5946] was to fix #739 "MassFlowSource_T with 2 ports does not balance flow among port[1]+port[2]" -- please don't revert this.

At the same time the temperature prescription was harmonized with the remaining fluid library, see e.g. Sources.Boundary_pT or Interfaces.PartialLumpedVolume in MSL 3.2 (and 3.2.1) -- it seems that the MassFlowSource models had been forgotten when introducing the dependence on Medium.ThermoStates in MSL 3.2. This is why I consider it a bug fix, even though the simulation results become slightly worse in this example.

modelica-trac-importer commented 7 years ago

Comment by msielemann on 24 May 2013 16:06 UTC Excellent, thanks for comments on Modelica.Fluid.Examples.Tanks.TanksWithOverflow

The obvious solution appears to be to ignore the difference during regression testing (see Hubertus remark on accuracy of of IF97)

modelica-trac-importer commented 7 years ago

Comment by hansolsson on 24 May 2013 16:14 UTC Just a minor note after seeing these messages:

Ignore gives the wrong impression for me - the intent is to update the reference to be the new result, and then use that as baseline for the regressions test. That way we avoid missing regressions between now and release of MSL 3.2.1.

I assume that is what people imply with "ignore", but just wanted to be clear.

modelica-trac-importer commented 7 years ago

Comment by mwetter on 24 May 2013 22:32 UTC I confirm that this is not a bug in MSL. If the length of the pipes are also changed in MSL 3.2 from 1 to 5 meters, then the results match. I recommend to accept the new results.

Replying to [comment:7 msielemann]:

Modelica.Fluid.Examples.TraceSubstances?.RoomCO2WithControls * Again, the change in simulation results identified during regression testing is not a bug in MSL * Instead, the length of two pipes was changed from 1m to 5m in [5925] * mwetter 15.01.2013 20:45:34
* Changed system.m_flow_nominal from 1 to 0.1, and eps_m_flow from ... * Reducing the pipe length back to 1m stalls the dassl integrator in Dymola 2013 FD01, which most like was the reason to increase it * Anyway, I suggest to ignore this difference during testing

modelica-trac-importer commented 7 years ago

Comment by rfranke on 25 May 2013 06:02 UTC ModelicaTest.Fluid.TestComponents.Machines.TestWaterPumpPowerCharacteristic

The difference is caused by the extended/fixed pump characteristics in MSL 3.2.1, see [5864] "Extend quadratic and polynomial pump curves with linear extrapolation", fixing #372. The simulation results deviate as the flow becomes negative at around 6s.

The new results should be accepted.

modelica-trac-importer commented 7 years ago

Comment by otter on 27 May 2013 09:24 UTC Replying to [comment:16 hansolsson]:

Just a minor note after seeing these messages:

Ignore gives the wrong impression for me - the intent is to update the reference to be the new result, and then use that as baseline for the regressions test. That way we avoid missing regressions between now and release of MSL 3.2.1.

I assume that is what people imply with "ignore", but just wanted to be clear.

It is not clear to me how to do this practically. In order to just use the regression setup from Dymola, one would have to re-introduce all changes to MSL 3.2 in the tagged version (or make a new branch from this tagged version and re-introduce the changes here). However, this can be quite involved because some changes are also due to other bug fixes in other components.

To summarize, in the next iteration regression failures from the above models are just ignored.

modelica-trac-importer commented 7 years ago

Comment by majetta on 27 May 2013 13:17 UTC Modelica.Electrical.Analog.Examples.CharacteristicThyristors

Modelica.Electrical.Analog.Examples.AmplifierWithOpAmpDetailed

modelica-trac-importer commented 7 years ago

Comment by rfranke on 1 Jun 2013 10:16 UTC ModelicaTest.Fluid.TestComponents.Vessels.TestTankWithTopPorts

See comments 10, 13, and 14 above.

The model uses two Modelica.Fluid.Sources.MassFlowSource_T with the StandardWater medium explicit in h. In MSL 3.2 this leads to two nonlinear equation systems to determine h from T. Considering Media.ThermoStates in MassFlowSource_T [5946], the implicit equation systems disappear from the solver. The specific enthalpy changes from 83.9161 kJ/kg in MSL 3.2 to 84.0131 kJ/kg in MSL 3.2.1.

The MassFlowSource_T and the TankWithTopPorts both use StandardWater and Medium.T_default.

The new unified treatment of Medium.ThermoStates leads to the constant value of 20.0232 °C for tank.medium.T during the simulation.

In MSL 3.2, ignoring Medium.ThermoStates in MassFlowSource_T, tank.medium.T gradually reduces from 20.0232 to 20.0007 during the simulation. This mixing/PT1 dynamics is wrong -- even though the final value is closer to Medium.T_default.

The new results should be accepted.

modelica-trac-importer commented 7 years ago

Comment by rfranke on 1 Jun 2013 11:06 UTC ModelicaTest.Fluid.TestComponents.Pipes.DynamicPipesAndFittings

The differences originate from [5845] "fix initial conditions and variants of dynamic pipe examples and tests".

First of all, the pipe model structures have been changed to avoid index reductions in connection sets, which led to overdetermined initial conditions for pressure states. Awaiting solution for #968.

At the same time the HeatTransferModels of pipe2, pipe5 and pipe9 have been unified, i.e. pipe5 was changed from LocalPipeFlowHeatTransfer to ConstantHeatTransfer. This explains the large difference for pipe7.mediums[5].h, compared to MSL3.2. The new value is identical to pipe11.mediums[5].h and slightly different from pipe1.mediums[5].h due to the additional junction volume. These new results appear more appropriate for a test that formulates the same model three times with slight variations.

The new results should be accepted.

modelica-trac-importer commented 7 years ago

Comment by rfranke on 1 Jun 2013 11:09 UTC ModelicaTest.Fluid.TestComponents.Pipes.DynamicPipesWithTraceSubstances

[5845] changed the durations of the ramps from 0 to 1s. At that time the duration 0 was not allowed (see #943). [6589] reverts this change.

Remaining differences are due to changed pipe model structures (awaiting solution for #968) and, possibly, trace substances.

modelica-trac-importer commented 7 years ago

Comment by otter on 26 Jul 2013 09:04 UTC Summary of the analysis of the different library officers:

Issues fixed or clarified:

Modelica.Blocks.Examples.BooleanNetwork Issue fixed.

Modelica.Electrical.Analog.Examples.CharacteristicThyristors Clarified. Changed example.

Modelica.Electrical.Analog.Examples.AmplifierWithOpAmpDetailed Very sensitive model. Issue no longer present in Dymola 2014 FD01 development.

Modelica.Electrical.Machines.Examples.AsynchronousInductionMachines.AIMC_withLosses Clarified. Different initial conditions.

Modelica.Electrical.Machines.Examples.DCMachines.DCPM_withLosses Issue fixed (wrong nominal value)

Modelica.Magnetic.FluxTubes.Examples.SolenoidActuator.ComparisonQuasiStationary Clarified. Improved model.

Modelica.Fluid.Examples.PumpingSystem Clarified. More detailed medium model.

Modelica.Fluid.Examples.HeatingSystem Clarified. Different initial conditions, different pipe structure.

Modelica.Fluid.Examples.Tanks.TanksWithOverflow Clarified. Reformulation with inverses and IF97 inverse not very precise.

Modelica.Fluid.Examples.AST_BatchPlant.BatchPlant_StandardWater Clarified. Improved pump models.

Modelica.Fluid.Examples.BranchingDynamicPipes Clarified. Different initial conditions.

Modelica.Fluid.Examples.HeatExchanger.HeatExchangerSimulation Clarified. Different initial conditions.

Modelica.Fluid.Examples.TraceSubstances.RoomCO2WithControls Clarified. Different parametrization.

Modelica.Thermal.FluidHeatFlow.Examples.PumpAndValve Issue fixed

ModelicaTest.Fluid.TestComponents.Machines.TestWaterPumpPowerCharacteristic Clarified. Improved pump models

ModelicaTest.Fluid.TestComponents.Pipes.DynamicPipesAndFittings Clarified. Changed model.

ModelicaTest.Fluid.TestComponents.Pipes.DynamicPipesWithTraceSubstances Clarified. Changed pipe model structure.

ModelicaTest.Fluid.TestComponents.Vessels.TestTankWithTopPorts Clarified. Reformulation with inverses and IF97 inverse not very precise.

Models that have not yet been analyzed:

ModelicaTest.Fluid.TestComponents.Fittings.TestJunctionTraceSubstances

ModelicaTest.Fluid.TestComponents.Fittings.TestMultiPortTraceSubstances

modelica-trac-importer commented 7 years ago

Comment by otter on 26 Jul 2013 14:01 UTC Performed a new regression testing with the following settings:

Reference: Modelica 3.2 (build 8, shipped with Dymola 2014 and earlier versions) ModelicaTest 3.2 (build 5, from svn) Dymola 2014

To be tested: Modelica 3.2.1 (build 1 in trunk) ModelicaTest 3.2.1 (build 1 in trunk) Dymola 2014 FD01 development

Result of the regression testing

Issues: Modelica.Magnetic.FundamentalWave.Examples.BasicMachines.AIMS_Start Fixed in add99e813b64e9b6b6ef1fece4d76f45d48f358f

ModelicaTest.Fluid.TestComponents.Vessels.TestInitialization Opened ticket #1222. Needs to be fixed.

Clarified from previous regression testing (differences due to different model/initialization)

Modelica.Electrical.Analog.Examples.CharacteristicThyristors Modelica.Electrical.Machines.Examples.AsynchronousInductionMachines.AIMC_withLosses Modelica.Magnetic.FluxTubes.Examples.SolenoidActuator.ComparisonQuasiStationary Modelica.Fluid.Examples.HeatingSystem Modelica.Fluid.Examples.Tanks.TanksWithOverflow Modelica.Fluid.Examples.BranchingDynamicPipes Modelica.Fluid.Examples.HeatExchanger.HeatExchangerSimulation Modelica.Fluid.Examples.TraceSubstances.RoomCO2WithControls ModelicaTest.Fluid.TestComponents.Machines.TestWaterPumpPowerCharacteristic ModelicaTest.Fluid.TestComponents.Pipes.DynamicPipesAndFittings ModelicaTest.Fluid.TestComponents.Pipes.DynamicPipesWithTraceSubstances ModelicaTest.Fluid.TestComponents.Vessels.TestTankWithTopPorts

Uncritical (minor differences) Modelica.Blocks.Examples.BooleanNetwork1 Variable rSFlipFlop.pre.u not found in reference file

Modelica.Electrical.Analog.Examples.HeatingNPN_OrGate Modelica.Electrical.Analog.Examples.AmplifierWithOpAmpDetailed Very small differences

Modelica.Electrical.Digital.Examples.Multiplexer Modelica.Electrical.Digital.Examples.FlipFlop Modelica.Electrical.Digital.Examples.HalfAdder Modelica.Electrical.Digital.Examples.FullAdder Modelica.Electrical.Digital.Examples.Adder4 Modelica.Electrical.Digital.Examples.Counter3 Modelica.Electrical.Digital.Examples.Counter New variables not found in reference file.

Modelica.Electrical.Machines.Examples.SynchronousInductionMachines.SMEE_LoadDump Very small differences

Modelica.Electrical.QuasiStationary.SinglePhase.Examples.Rectifier Initial conditions for pre(x) of continuous-time variable x missing. Dymola selects by default the right initial conditions x=0. In Modelica 3.2 rev 2 this case was clarified that for continuous-time variables x the initial equation "pre(x) = x" is automatically introduced.

Modelica.Electrical.Spice3.Examples.Oscillator Modelica.Magnetic.FluxTubes.Examples.SolenoidActuator.ComparisonPullInStroke: New variables not found in reference file.

Modelica.Mechanics.MultiBody.Examples.Elementary.RollingWheelSetPulling Very small differences.

Modelica.Mechanics.Translational.Examples.HeatLosses Different models due to a bug fix (brake + mass with friction directly coupled)

Modelica.Fluid.Examples.PumpingSystem Modelica.Fluid.Examples.AST_BatchPlant.BatchPlant_StandardWater Initial conditions for pre(y) of continuous-time variable y missing. Dymola does not yet support the recently introduced clarification that for continuous-time variables y the initial equation "pre(y) = y" is automatically introduced (Dymola introduces a default start variable, but in pedantic mode this gives an error).

Modelica.Fluid.Examples.IncompressibleFluidNetwork Quite different models (heating source, different step startTime, different end time)

Modelica.Media.Examples.TwoPhaseWater.TestTwoPhaseStates Very small differences.

Modelica.Thermal.HeatTransfer.Examples.Motor ModelicaTest.Blocks.Limiters ModelicaTest.Tables.CombiTimeTable.TestXX (about 60 test cases) New variable not found in reference file.

ModelicaTest.Fluid.TestComponents.Fittings.TestJunctionTraceSubstances Small difference for concentration.

ModelicaTest.Fluid.TestComponents.Fittings.TestMultiPortTraceSubstances Different initial conditions for the concentration (MSL 3.2: c=0; MSL 3.2.1: c=1e-4)

ModelicaTest.Fluid.TestComponents.Machines.TestWaterPumpDCMotor ModelicaTest.Fluid.TestComponents.Machines.TestWaterPumpDCMotorHeatTransfer ModelicaTest.MultiBody.Parts.Rotor1D.Rotor1D_2 ModelicaTest.MultiBody.Parts.Rotor1D.Rotor1D_3 Very small differences.

modelica-trac-importer commented 7 years ago

Comment by otter on 31 Jul 2013 18:57 UTC Performed a third regression testing with MSL 3.2.1 build 1 (RC1):

Same result as above with the following exceptions:

Modelica.Electrical.Machines.Examples.SynchronousInductionMachines.SMPM_Inverter Modelica.Electrical.Machines.Examples.SynchronousInductionMachines.SMEE_Generator Modelica.Electrical.Machines.Examples.SynchronousInductionMachines.SMEE_LoadDump Modelica.Magnetic.FundamentalWave.Examples.BasicMachines.AIMS_Start Modelica.Magnetic.FundamentalWave.Examples.BasicMachines.SMPM_Inverter Modelica.Magnetic.FundamentalWave.Examples.BasicMachines.SMEE_Generator

In all the models above, some internal currents have a negative sign as compared to the reference. The reason is the bug-fix #1226.

To summarize: Regression testing is successful.