Missing features to test the power grids benchmark in MARCO

casella commented 6 months ago

To my knowlege, these issues must be resolved

[x] create a version of the benchmark that doesn't have for-loops with if's. This is not really how real models are built. The benchmark should rather be generated by a script with scalar instances (as in the ScalableTestGrids library) and then those instances should be collected by the OMC frontend into homogeneous arrays
[x] resolve the issue of start attributes in record instances OpenModelica/OpenModelica#11792
[x] fixing issues with record flattening OpenModelica/OpenModelica#12490
[x] add the parsing of such start attributes in MARCO

The IDA solver of MARCO has a known issue with the sum() reduction operator, which currently does not generate the right Jacobian for the solver. However, this operator is not used by the benchmark, so everything should be ready once the above-mentioned issues are resolved.

mscuttari commented 5 months ago

After closer inspection, I realized that MARCO already supports start attributes of record attributes, as visible in this test. The sum reduction operator is also supported using the IDA solver, as per #15.

casella commented 5 months ago

The OO version has been there for a while already, see Benchmarks/PowerGrid/Modelica_OO 😅

mscuttari commented 5 months ago

The output of OMC (v1.24.0-dev-148-gac439f6130-cmake) when processing the builder model seems quite broken. For example, the command

omc -i=PowerGridOOModelBuilders.ModelBuilder_Ne_2 PowerGridOOModelBuilders.mo --baseModelica -d=nonfScalarize,mergeComponents,combineSubscripts,evaluateAllParameters,vectorizeBindings

leads to

package 'ModelBuilder_Ne_2'
  function 'Modelica.Utilities.Files.fullPathName' "Get full path name of file or directory name"
    input String 'name';
    output String 'fullName';
  external "C" 'fullName' = ModelicaInternal_fullPathName('name') annotation(Library = "ModelicaExternalC", LibraryDirectory = "modelica://Modelica/Resources/Library", Include = "#include \"ModelicaInternal.h\"", IncludeDirectory = "modelica://Modelica/Resources/C-Sources");
  end 'Modelica.Utilities.Files.fullPathName';

  function 'Modelica.Utilities.Files.remove' "Remove file or directory (ignore call, if it does not exist)"
    input String 'name';
    String 'fullName';
    'Modelica.Utilities.Types.FileType' 'fileType' = 'Modelica.Utilities.Internal.FileSystem.stat'('name');
  algorithm
    if 'fileType' == 'Modelica.Utilities.Types.FileType'.RegularFile or 'fileType' == 'Modelica.Utilities.Types.FileType'.SpecialFile then
      'Modelica.Utilities.Internal.FileSystem.removeFile'('name');
    elseif 'fileType' == 'Modelica.Utilities.Types.FileType'.Directory then
      'fullName' := 'Modelica.Utilities.Files.fullPathName'('name');
      'Modelica.Utilities.Files.remove.removeDirectory'('fullName');
    end if;
  end 'Modelica.Utilities.Files.remove';

  function 'Modelica.Utilities.Files.remove.removeDirectory' "Remove a directory, even if it is not empty"
    input String 'name';
    Integer 'lenName' = 'Modelica.Utilities.Strings.length'('name');
    String 'name2' = if 'Modelica.Utilities.Strings.substring'('name', 'lenName', 'lenName') == "/" then 'Modelica.Utilities.Strings.substring'('name', 'lenName' - 1, 'lenName' - 1) else 'name';
  algorithm
    'Modelica.Utilities.Files.remove.removeDirectoryContents'('Modelica.Utilities.Internal.FileSystem.readDirectory'('name2', 'Modelica.Utilities.Internal.FileSystem.getNumberOfFiles'('name2')), 'name2');
    'Modelica.Utilities.Internal.FileSystem.rmdir'('name2');
  end 'Modelica.Utilities.Files.remove.removeDirectory';

  function 'Modelica.Utilities.Files.remove.removeDirectoryContents'
    input String[:] 'fileNames';
    input String 'name2';
  algorithm
    for 'i' in 1:size('fileNames', 1) loop
      'Modelica.Utilities.Files.remove'('name2' + "/" + 'fileNames'['i']);
    end for;
  end 'Modelica.Utilities.Files.remove.removeDirectoryContents';

  function 'Modelica.Utilities.Internal.FileSystem.getNumberOfFiles' "Get number of files and directories in a directory (POSIX functions opendir, readdir, closedir)"
    input String 'directory';
    output Integer 'result';
  external "C" 'result' = ModelicaInternal_getNumberOfFiles('directory') annotation(Library = "ModelicaExternalC", LibraryDirectory = "modelica://Modelica/Resources/Library", Include = "#include \"ModelicaInternal.h\"", IncludeDirectory = "modelica://Modelica/Resources/C-Sources");
  end 'Modelica.Utilities.Internal.FileSystem.getNumberOfFiles';

  function 'Modelica.Utilities.Internal.FileSystem.readDirectory' "Read names of a directory (POSIX functions opendir, readdir, closedir)"
    input String 'directory';
    input Integer 'nNames';
    output String['nNames'] 'names';
  external "C" ModelicaInternal_readDirectory('directory', 'nNames', 'names') annotation(Library = "ModelicaExternalC", LibraryDirectory = "modelica://Modelica/Resources/Library", Include = "#include \"ModelicaInternal.h\"", IncludeDirectory = "modelica://Modelica/Resources/C-Sources");
  end 'Modelica.Utilities.Internal.FileSystem.readDirectory';

  function 'Modelica.Utilities.Internal.FileSystem.removeFile' "Remove existing file (C function 'remove')"
    input String 'fileName';
  external "C" ModelicaInternal_removeFile('fileName') annotation(Library = "ModelicaExternalC", LibraryDirectory = "modelica://Modelica/Resources/Library", Include = "#include \"ModelicaInternal.h\"", IncludeDirectory = "modelica://Modelica/Resources/C-Sources");
  end 'Modelica.Utilities.Internal.FileSystem.removeFile';

  function 'Modelica.Utilities.Internal.FileSystem.rmdir' "Remove empty directory (POSIX function 'rmdir')"
    input String 'directoryName';
  external "C" ModelicaInternal_rmdir('directoryName') annotation(Library = "ModelicaExternalC", LibraryDirectory = "modelica://Modelica/Resources/Library", Include = "#include \"ModelicaInternal.h\"", IncludeDirectory = "modelica://Modelica/Resources/C-Sources");
  end 'Modelica.Utilities.Internal.FileSystem.rmdir';

  function 'Modelica.Utilities.Internal.FileSystem.stat' "Inquire file information (POSIX function 'stat')"
    input String 'name';
    output 'Modelica.Utilities.Types.FileType' 'fileType';
  external "C" 'fileType' = ModelicaInternal_stat('name') annotation(Library = "ModelicaExternalC", LibraryDirectory = "modelica://Modelica/Resources/Library", Include = "#include \"ModelicaInternal.h\"", IncludeDirectory = "modelica://Modelica/Resources/C-Sources");
  end 'Modelica.Utilities.Internal.FileSystem.stat';

  function 'Modelica.Utilities.Strings.length' "Return length of string"
    input String 'string';
    output Integer 'result';
  external "C" 'result' = ModelicaStrings_length('string') annotation(Library = "ModelicaExternalC", LibraryDirectory = "modelica://Modelica/Resources/Library", Include = "#include \"ModelicaStrings.h\"", IncludeDirectory = "modelica://Modelica/Resources/C-Sources");
  end 'Modelica.Utilities.Strings.length';

  function 'Modelica.Utilities.Strings.substring' "Return a substring defined by start and end index"
    input String 'string';
    input Integer 'startIndex'(min = 1);
    input Integer 'endIndex'(min = 1);
    output String 'result';
  external "C" 'result' = ModelicaStrings_substring('string', 'startIndex', 'endIndex') annotation(Library = "ModelicaExternalC", LibraryDirectory = "modelica://Modelica/Resources/Library", Include = "#include \"ModelicaStrings.h\"", IncludeDirectory = "modelica://Modelica/Resources/C-Sources");
  end 'Modelica.Utilities.Strings.substring';

  function 'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'
    input String 'string' = "";
    input String 'fileName' = "";
  external "C" ModelicaInternal_print('string', 'fileName') annotation(Library = "ModelicaExternalC", LibraryDirectory = "modelica://PowerGridOOModelBuilders/Resources/Library", Include = "#include \"ModelicaInternal.h\"", IncludeDirectory = "modelica://Modelica/Resources/C-Sources");
  end 'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print';

  type 'Modelica.Utilities.Types.FileType' = enumeration(NoFile, RegularFile, Directory, SpecialFile);

  model 'ModelBuilder_Ne_2'
    parameter Integer 'Ne' = 2 "Number of even rows and columns of the grid";
    final parameter Integer 'N' = 4 "Number of even rows and columns of the grid";
    final parameter Integer 'Ng' = 2 "Number of generators on each row and column";
    String 'f' = "D:/Temp/GridModels/Grid_Ne_2.mo";
  algorithm
    when initial() then
      'Modelica.Utilities.Files.remove'('f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("model Grid_Ne_2", 'f');

      for 'i' in 1:4 loop
        for 'j' in 1:4 loop
          'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  PowerGridOO.Components.Bus bus_" + String('i', 0, true) + "_" + String('j', 0, true) + ";", 'f');
        end for;
      end for;

      for 'i' in 1:4 loop
        for 'j' in 1:2 loop
          'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  PowerGridOO.Components.Generator gen_" + String('i', 0, true) + "_" + String('j', 0, true) + ";", 'f');
        end for;
      end for;

      for 'i' in 1:4 loop
        for 'j' in 1:2 loop
          'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  PowerGridOO.Components.Load load_" + String('i', 0, true) + "_" + String('j', 0, true) + ";", 'f');
        end for;
      end for;

      for 'i' in 1:4 loop
        for 'j' in 1:3 loop
          'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  PowerGridOO.Components.Line line_h_" + String('i', 0, true) + "_" + String('j', 0, true) + ";", 'f');
        end for;
      end for;

      for 'i' in 1:3 loop
        for 'j' in 1:4 loop
          'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  PowerGridOO.Components.Line line_v_" + String('i', 0, true) + "_" + String('j', 0, true) + ";", 'f');
        end for;
      end for;

      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  output PowerGridOO.Types.ComplexPU v_out[4];", 'f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  output PowerGridOO.Types.PU omega_out[4];", 'f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("equation", 'f');

      for 'i' in 1:2:4 loop
        for 'j' in 1:2 loop
          'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  connect(gen_" + String('i', 0, true) + "_" + String('j', 0, true) + ".port, bus_" + String('i', 0, true) + "_" + String(2 * 'j' - 1, 0, true) + ".port);", 'f');
        end for;
      end for;

      for 'i' in 2:2:4 loop
        for 'j' in 1:2 loop
          'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  connect(gen_" + String('i', 0, true) + "_" + String('j', 0, true) + ".port, bus_" + String('i', 0, true) + "_" + String(2 * 'j', 0, true) + ".port);", 'f');
        end for;
      end for;

      for 'i' in 1:2:4 loop
        for 'j' in 1:2 loop
          'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  connect(load_" + String('i', 0, true) + "_" + String('j', 0, true) + ".port, bus_" + String('i', 0, true) + "_" + String(2 * 'j', 0, true) + ".port);", 'f');
        end for;
      end for;

      for 'i' in 2:2:4 loop
        for 'j' in 1:2 loop
          'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  connect(load_" + String('i', 0, true) + "_" + String('j', 0, true) + ".port, bus_" + String('i', 0, true) + "_" + String(2 * 'j' - 1, 0, true) + ".port);", 'f');
        end for;
      end for;

      for 'i' in 1:4 loop
        for 'j' in 1:3 loop
          'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  connect(line_h_" + String('i', 0, true) + "_" + String('j', 0, true) + ".portA, bus_" + String('i', 0, true) + "_" + String('j', 0, true) + ".port);", 'f');
          'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  connect(line_h_" + String('i', 0, true) + "_" + String('j', 0, true) + ".portB, bus_" + String('i', 0, true) + "_" + String('j' + 1, 0, true) + ".port);", 'f');
        end for;
      end for;

      for 'i' in 1:3 loop
        for 'j' in 1:4 loop
          'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  connect(line_v_" + String('i', 0, true) + "_" + String('j', 0, true) + ".portA, bus_" + String('i', 0, true) + "_" + String('j', 0, true) + ".port);", 'f');
          'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  connect(line_v_" + String('i', 0, true) + "_" + String('j', 0, true) + ".portB, bus_" + String('i' + 1, 0, true) + "_" + String('j', 0, true) + ".port);", 'f');
        end for;
      end for;

      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  v_out[1] = bus_1_1.port.v;", 'f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  v_out[2] = bus_1_4.port.v;", 'f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  v_out[3] = bus_4_1.port.v;", 'f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  v_out[4] = bus_4_4.port.v;", 'f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  omega_out[1] = gen_1_1.omega;", 'f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  omega_out[2] = gen_1_2.omega;", 'f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  omega_out[3] = gen_4_1.omega;", 'f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  omega_out[4] = gen_4_2.omega;", 'f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("  annotation(__OpenModelica_commandLineOptions = \"-d=execstat --daeMode --tearingMethod=minimalTearing\",", 'f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("             __OpenModelica_simulationFlags(nls=\"kinsol\", lv=\"LOG_STATS\", noEquidistantTimeGrid = \"()\"),", 'f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("             experiment(StopTime = 5, Tolerance = 1e-6));", 'f');
      'PowerGridOOModelBuilders.ModelBuilder_Ne_2.print'("end Grid_Ne_2;", 'f');
    end when;
  end 'ModelBuilder_Ne_2';
end 'ModelBuilder_Ne_2';

Maybe am I not understanding how to use it?

casella commented 5 months ago

The idea is to run the ModelBuilder models in OMC to generate the source code of the tests, add them to the library, and then running them with MARCO. You can already run, e.g. PowerGridOO.Test.Grid_Ne_1 or PowerGridOO.Test.Grid_Ne_2.

But first, we need to sort out the flat Modelica output from OMC, there are still a couple of open issues.

Question: can MARCO handle operator records?

casella commented 5 months ago

The issues in OpenModelica/OpenModelica#12490 should be resolved now, I'm building a new Windows nightly build to test them. This will require some changes in the flags, w.r.t. what we have now in OMC.md.

With a bit of luck everything should be ready for today's Modelica meeting.

casella commented 5 months ago

Regarding OpenModelica/OpenModelica#11792, if MARCO can handle Complex operator records natively, everything should be now OK, otherwise we still need to do something on the OMC BaseModelica output front.

mscuttari commented 5 months ago

OMC may need further works. I tested Grid_Ne_1 and the dimensions of variables are wrong:

  model 'Grid_Ne_1'
    ...
    'PowerGridOO.Types.ComplexPU' '$Line1.portA.v'('re'(start = fill(1.0, 4))) "Voltage phasor in p.u.";
    ...

$Line1.portA.v is typed as a record but should be an array of records.

Reproducing command: omc -i=PowerGridOO.Test.Grid_Ne_1 PowerGridOO.mo --baseModelica -d=nonfScalarize,mergeComponents,combineSubscripts,evaluateAllParameters,vectorizeBindings

casella commented 5 months ago

Sure, we need those two commits to be merged in and probably also to change the magic incantations.

casella commented 5 months ago

See OMC.md.

I tried the flat Grid_Ne_1.mo model obtained with OMC and the latest flags, but I got some invalid C code, see OpenModelica/OpenModelica#12598. This may not be a problem for MARCO, maybe it's just an OMC glitch. I tried to run it in Dymola, it compiles correctly, but then the runtime complains it is singular, while it can run the original model without problems. I'll need to check if the equations are different, though it's a bit boring, since there are 84 equations in that model...

marco-compiler / marco

Missing features to test the power grids benchmark in MARCO #14