Question about Enforcement of Boundary Conditions/Metis Partitioning for SU2/Unstructured Grids and Further Documentation for Steady-State Solver for Eilmer v4

[plaad]

Hi All,

I was interested in using Eilmer v4 for research at my university as it has many great features. I have since worked through various examples.

I have set up a small, simple supersonic test case based on the unstructured grid SU2 examples (single block/domain 2D box with an inflow (left), outflow (right), wall (bottom), and symmetry (top) BCs) in which I perform Metis partitioning for 1 core.

1) When using the transient solver, the run proceeds and the initial flow conditions seem to initialize to the correct specified values for the variables, but my BCs seem to not quite be correct.

In addition, I compared the original grid processed by the ugrid_partition command (test1.su2) into the partitioned block (block_0_test1.su2) in su2-grid folder and noticed the partition grid does not seem to preserve the same information of the original grid as performing the diff block_0_test1.su2 test1.su2 -y command on the two files shows the BCs elements/faces for the BCs as shown:

---

% | NDIME= 2 % Problem dimension | NELEM= 81 % | 9 0 4 36 19 0 NDIME= 2 | 9 19 36 37 18 1 % | 9 18 37 38 17 2 % Inner element connectivity | 9 17 38 39 16 3 % | 9 16 39 40 15 4 NELEM= 81 | 9 15 40 41 14 5 9 0 1 2 3 0 | 9 14 41 42 13 6 9 3 2 4 5 1 | 9 13 42 43 12 7 9 5 4 6 7 2 | 9 12 43 27 2 8 9 7 6 8 9 3 | 9 4 5 44 36 9 9 9 8 10 11 4 | 9 36 44 45 37 10 9 11 10 12 13 5 | 9 37 45 46 38 11 9 13 12 14 15 6 | 9 38 46 47 39 12 9 15 14 16 17 7 | 9 39 47 48 40 13 9 17 16 18 19 8 | 9 40 48 49 41 14 9 1 20 21 2 9 | 9 41 49 50 42 15 9 2 21 22 4 10 | 9 42 50 51 43 16 9 4 22 23 6 11 | 9 43 51 26 27 17 9 6 23 24 8 12 | 9 5 6 52 44 18 9 8 24 25 10 13 | 9 44 52 53 45 19 9 10 25 26 12 14 | 9 45 53 54 46 20 9 12 26 27 14 15 | 9 46 54 55 47 21 9 14 27 28 16 16 | 9 47 55 56 48 22 9 16 28 29 18 17 | 9 48 56 57 49 23 9 20 30 31 21 18 | 9 49 57 58 50 24 9 21 31 32 22 19 | 9 50 58 59 51 25 9 22 32 33 23 20 | 9 51 59 25 26 26 9 23 33 34 24 21 | 9 6 7 60 52 27 9 24 34 35 25 22 | 9 52 60 61 53 28 9 25 35 36 26 23 | 9 53 61 62 54 29 9 26 36 37 27 24 | 9 54 62 63 55 30 9 27 37 38 28 25 | 9 55 63 64 56 31 9 28 38 39 29 26 | 9 56 64 65 57 32 9 30 40 41 31 27 | 9 57 65 66 58 33 9 31 41 42 32 28 | 9 58 66 67 59 34 9 32 42 43 33 29 | 9 59 67 24 25 35 9 33 43 44 34 30 | 9 7 8 68 60 36 9 34 44 45 35 31 | 9 60 68 69 61 37 9 35 45 46 36 32 | 9 61 69 70 62 38 9 36 46 47 37 33 | 9 62 70 71 63 39 9 37 47 48 38 34 | 9 63 71 72 64 40 9 38 48 49 39 35 | 9 64 72 73 65 41 9 40 50 51 41 36 | 9 65 73 74 66 42 9 41 51 52 42 37 | 9 66 74 75 67 43 9 42 52 53 43 38 | 9 67 75 23 24 44 9 43 53 54 44 39 | 9 8 9 76 68 45 9 44 54 55 45 40 | 9 68 76 77 69 46 9 45 55 56 46 41 | 9 69 77 78 70 47 9 46 56 57 47 42 | 9 70 78 79 71 48 9 47 57 58 48 43 | 9 71 79 80 72 49 9 48 58 59 49 44 | 9 72 80 81 73 50 9 50 60 61 51 45 | 9 73 81 82 74 51 9 51 61 62 52 46 | 9 74 82 83 75 52 9 52 62 63 53 47 | 9 75 83 22 23 53 9 53 63 64 54 48 | 9 9 10 84 76 54 9 54 64 65 55 49 | 9 76 84 85 77 55 9 55 65 66 56 50 | 9 77 85 86 78 56 9 56 66 67 57 51 | 9 78 86 87 79 57 9 57 67 68 58 52 | 9 79 87 88 80 58 9 58 68 69 59 53 | 9 80 88 89 81 59 9 60 70 71 61 54 | 9 81 89 90 82 60 9 61 71 72 62 55 | 9 82 90 91 83 61 9 62 72 73 63 56 | 9 83 91 21 22 62 9 63 73 74 64 57 | 9 10 11 92 84 63 9 64 74 75 65 58 | 9 84 92 93 85 64 9 65 75 76 66 59 | 9 85 93 94 86 65 9 66 76 77 67 60 | 9 86 94 95 87 66 9 67 77 78 68 61 | 9 87 95 96 88 67 9 68 78 79 69 62 | 9 88 96 97 89 68 9 70 80 81 71 63 | 9 89 97 98 90 69 9 71 81 82 72 64 | 9 90 98 99 91 70 9 72 82 83 73 65 | 9 91 99 20 21 71 9 73 83 84 74 66 | 9 11 1 28 92 72 9 74 84 85 75 67 | 9 92 28 29 93 73 9 75 85 86 76 68 | 9 93 29 30 94 74 9 76 86 87 77 69 | 9 94 30 31 95 75 9 77 87 88 78 70 | 9 95 31 32 96 76 9 78 88 89 79 71 | 9 96 32 33 97 77 9 80 90 91 81 72 | 9 97 33 34 98 78 9 81 91 92 82 73 | 9 98 34 35 99 79 9 82 92 93 83 74 | 9 99 35 3 20 80 9 83 93 94 84 75 | NPOIN= 100 9 84 94 95 85 76 | 0.5 0 0 9 85 95 96 86 77 | -0.5 0 1 9 86 96 97 87 78 | 0.5 1 2 9 87 97 98 88 79 | -0.5 1 3 9 88 98 99 89 80 | 0.388888888888889 0 4 % | 0.2777777777777778 0 5 % Node coordinates | 0.1666666666666667 0 6 % | 0.05555555555555569 0 7 NPOIN= 100 | -0.05555555555555555 0 8 0.50000000000000000 0.00000000000000000 0 | -0.1666666666666666 0 9 0.38888888888888901 0.00000000000000000 1 | -0.2777777777777777 0 10 0.38888888888888901 0.00999999782105232 2 | -0.3888888888888888 0 11 0.50000000000000000 0.00999999730949401 3 | 0.5 0.6304968550065105 12 0.38888888888888890 0.02570189887168589 4 | 0.5 0.3951730439281571 13 0.50000000000000000 0.02570189817079858 5 | 0.5 0.2453034246537306 14 0.38888888888888878 0.05035687038068393 6 | 0.5 0.1498566368955266 15 0.50000000000000000 0.05035686944087683 7 | 0.5 0.08906986681414855 16 0.38888888888888890 0.08906986818728019 8 | 0.5 0.05035686944087683 17 0.50000000000000000 0.08906986681414855 9 | 0.5 0.02570189817079858 18 0.38888888888888878 0.14985663829703441 10 | 0.5 0.009999997309494007 19 0.50000000000000000 0.14985663689552661 11 | -0.388888888888889 1 20 0.38888888888888890 0.24530342653583179 12 | -0.2777777777777778 1 21 0.50000000000000000 0.24530342465373059 13 | -0.1666666666666667 1 22 0.38888888888888878 0.39517304580465362 14 | -0.05555555555555569 1 23 0.50000000000000000 0.39517304392815711 15 | 0.05555555555555555 1 24 0.38888888888888878 0.63049685619459150 16 | 0.1666666666666666 1 25 0.50000000000000000 0.63049685500651054 17 | 0.2777777777777777 1 26 0.38888888888888878 1.00000000000000000 18 | 0.3888888888888888 1 27 0.50000000000000000 1.00000000000000000 19 | -0.5 0.01000000191351902 28 0.27777777777777779 0.00000000000000000 20 | -0.5 0.02570190447878462 29 0.27777777777777779 0.00999999833261067 21 | -0.5 0.05035687789914145 30 0.27777777777777779 0.02570189957257324 22 | -0.5 0.08906987917233344 31 0.27777777777777779 0.05035687132049116 23 | -0.5 0.1498566495090978 32 0.27777777777777779 0.08906986956041180 24 | -0.5 0.2453034415926399 33 0.27777777777777768 0.14985663969854221 25 | -0.5 0.3951730608166252 34 0.27777777777777768 0.24530342841793271 26 | -0.5 0.6304968656992394 35 0.27777777777777768 0.39517304768115002 27 | 0.388888888888889 0.00999999782105232 36 0.27777777777777768 0.63049685738267247 28 | 0.3888888888888889 0.02570189887168589 37 0.27777777777777768 1.00000000000000000 29 | 0.3888888888888888 0.05035687038068393 38 0.16666666666666671 0.00000000000000000 30 | 0.3888888888888889 0.08906986818728019 39 0.16666666666666680 0.00999999884416908 31 | 0.3888888888888888 0.1498566382970344 40 0.16666666666666671 0.02570190027346064 32 | 0.3888888888888889 0.2453034265358318 41 0.16666666666666680 0.05035687226029834 33 | 0.3888888888888888 0.3951730458046536 42 0.16666666666666671 0.08906987093354346 34 | 0.3888888888888888 0.6304968561945915 43 0.16666666666666671 0.14985664110005029 35 | 0.2777777777777778 0.009999998332610671 44 0.16666666666666671 0.24530343030003390 36 | 0.2777777777777778 0.02570189957257324 45 0.16666666666666671 0.39517304955764643 37 | 0.2777777777777778 0.05035687132049116 46 0.16666666666666671 0.63049685857075344 38 | 0.2777777777777778 0.0890698695604118 47 0.16666666666666660 1.00000000000000000 39 | 0.2777777777777777 0.1498566396985422 48 0.05555555555555569 0.00000000000000000 40 | 0.2777777777777777 0.2453034284179327 49 0.05555555555555568 0.00999999935572737 41 | 0.2777777777777777 0.39517304768115 50 0.05555555555555566 0.02570190097434794 42 | 0.2777777777777777 0.6304968573826725 51 0.05555555555555566 0.05035687320010557 43 | 0.1666666666666668 0.009999998844169078 52 0.05555555555555568 0.08906987230667524 44 | 0.1666666666666667 0.02570190027346064 53 0.05555555555555570 0.14985664250155820 45 | 0.1666666666666668 0.05035687226029834 54 0.05555555555555566 0.24530343218213480 46 | 0.1666666666666667 0.08906987093354346 55 0.05555555555555565 0.39517305143414289 47 | 0.1666666666666667 0.1498566411000503 56 0.05555555555555558 0.63049685975883452 48 | 0.1666666666666667 0.2453034303000339 57 0.05555555555555555 1.00000000000000000 49 | 0.1666666666666667 0.3951730495576464 58 -0.05555555555555555 0.00000000000000000 50 | 0.1666666666666667 0.6304968585707534 59 -0.05555555555555555 0.00999999986728572 51 | 0.05555555555555568 0.009999999355727374 60 -0.05555555555555552 0.02570190167523534 52 | 0.05555555555555566 0.02570190097434794 61 -0.05555555555555553 0.05035687413991274 53 | 0.05555555555555566 0.05035687320010557 62 -0.05555555555555559 0.08906987367980679 54 | 0.05555555555555568 0.08906987230667524 63 -0.05555555555555559 0.14985664390306611 55 | 0.0555555555555557 0.1498566425015582 64 -0.05555555555555557 0.24530343406423591 56 | 0.05555555555555566 0.2453034321821348 65 -0.05555555555555562 0.39517305331063929 57 | 0.05555555555555565 0.3951730514341429 66 -0.05555555555555561 0.63049686094691548 58 | 0.05555555555555558 0.6304968597588345 67 -0.05555555555555569 1.00000000000000000 59 | -0.05555555555555555 0.009999999867285725 68 -0.16666666666666660 0.00000000000000000 60 | -0.05555555555555552 0.02570190167523534 69 -0.16666666666666671 0.01000000037884397 61 | -0.05555555555555553 0.05035687413991274 70 -0.16666666666666671 0.02570190237612263 62 | -0.05555555555555559 0.08906987367980679 71 -0.16666666666666660 0.05035687507971981 63 | -0.05555555555555559 0.1498566439030661 72 -0.16666666666666671 0.08906987505293845 64 | -0.05555555555555557 0.2453034340642359 73 -0.16666666666666671 0.14985664530457399 65 | -0.05555555555555562 0.3951730533106393 74 -0.16666666666666660 0.24530343594633691 66 | -0.05555555555555561 0.6304968609469155 75 -0.16666666666666660 0.39517305518713580 67 | -0.1666666666666667 0.01000000037884397 76 -0.16666666666666671 0.63049686213499645 68 | -0.1666666666666667 0.02570190237612263 77 -0.16666666666666671 1.00000000000000000 69 | -0.1666666666666666 0.05035687507971981 78 -0.27777777777777768 0.00000000000000000 70 | -0.1666666666666667 0.08906987505293845 79 -0.27777777777777762 0.01000000089040248 71 | -0.1666666666666667 0.149856645304574 80 -0.27777777777777768 0.02570190307701004 72 | -0.1666666666666666 0.2453034359463369 81 -0.27777777777777779 0.05035687601952699 73 | -0.1666666666666666 0.3951730551871358 82 -0.27777777777777768 0.08906987642607023 74 | -0.1666666666666667 0.6304968621349964 83 -0.27777777777777768 0.14985664670608190 75 | -0.2777777777777776 0.01000000089040248 84 -0.27777777777777762 0.24530343782843789 76 | -0.2777777777777777 0.02570190307701004 85 -0.27777777777777768 0.39517305706363232 77 | -0.2777777777777778 0.05035687601952699 86 -0.27777777777777768 0.63049686332307742 78 | -0.2777777777777777 0.08906987642607023 87 -0.27777777777777779 1.00000000000000000 79 | -0.2777777777777777 0.1498566467060819 88 -0.38888888888888878 0.00000000000000000 80 | -0.2777777777777776 0.2453034378284379 89 -0.38888888888888890 0.01000000140196072 81 | -0.2777777777777777 0.3951730570636323 90 -0.38888888888888878 0.02570190377789727 82 | -0.2777777777777777 0.6304968633230774 91 -0.38888888888888878 0.05035687695933422 83 | -0.3888888888888889 0.01000000140196072 92 -0.38888888888888878 0.08906987779920184 84 | -0.3888888888888888 0.02570190377789727 93 -0.38888888888888878 0.14985664810758989 85 | -0.3888888888888888 0.05035687695933422 94 -0.38888888888888878 0.24530343971053889 86 | -0.3888888888888888 0.08906987779920184 95 -0.38888888888888901 0.39517305894012872 87 | -0.3888888888888888 0.1498566481075899 96 -0.38888888888888890 0.63049686451115838 88 | -0.3888888888888888 0.2453034397105389 97 -0.38888888888888901 1.00000000000000000 89 | -0.388888888888889 0.3951730589401287 98 -0.50000000000000000 0.00000000000000000 90 | -0.3888888888888889 0.6304968645111584 99 -0.50000000000000000 0.01000000191351902 91 | NMARK= 5 -0.50000000000000000 0.02570190447878462 92 | MARKER_TAG= WALLS -0.50000000000000000 0.05035687789914145 93 | MARKER_ELEMS= 9 -0.50000000000000000 0.08906987917233344 94 | 3 0 4 -0.50000000000000000 0.14985664950909780 95 | 3 4 5 -0.50000000000000000 0.24530344159263989 96 | 3 5 6 -0.50000000000000000 0.39517306081662518 97 | 3 6 7 -0.50000000000000000 0.63049686569923935 98 | 3 7 8 -0.50000000000000000 1.00000000000000000 99 | 3 8 9 % | 3 9 10 % Boundary elements | 3 10 11 % | 3 11 1 NMARK= 4 | MARKER_TAG= INFLOW MARKER_TAG= WALLS | MARKER_ELEMS= 9 MARKER_ELEMS= 9 | 3 1 28 3 0 1 | 3 28 29 3 1 20 | 3 29 30 3 20 30 | 3 30 31 3 30 40 | 3 31 32 3 40 50 | 3 32 33 3 50 60 | 3 33 34 3 60 70 | 3 34 35 3 70 80 | 3 35 3 3 80 90 | MARKER_TAG= OUTFLOW MARKER_TAG= INFLOW | MARKER_ELEMS= 9 MARKER_ELEMS= 9 | 3 2 12 3 90 91 | 3 12 13 3 91 92 | 3 13 14 3 92 93 | 3 14 15 3 93 94 | 3 15 16 3 94 95 | 3 16 17 3 95 96 | 3 17 18 3 96 97 | 3 18 19 3 97 98 | 3 19 0 3 98 99 | MARKER_TAG= SYMMETRY MARKER_TAG= OUTFLOW | MARKER_ELEMS= 9 MARKER_ELEMS= 9 | 3 3 20 3 3 0 | 3 20 21 3 5 3 | 3 21 22 3 7 5 | 3 22 23 3 9 7 | 3 23 24 3 11 9 | 3 24 25 3 13 11 | 3 25 26 3 15 13 | 3 26 27 3 17 15 | 3 27 2 3 19 17 | MARKER_TAG= FLUID MARKER_TAG= SYMMETRY | MARKER_ELEMS= 81 MARKER_ELEMS= 9 | 9 0 4 36 19 3 18 19 | 9 19 36 37 18 3 29 18 | 9 18 37 38 17 3 39 29 | 9 17 38 39 16 3 49 39 | 9 16 39 40 15 3 59 49 | 9 15 40 41 14 3 69 59 | 9 14 41 42 13 3 79 69 | 9 13 42 43 12 3 89 79 | 9 12 43 27 2 3 99 89 | 9 4 5 44 36

9 36 44 45 37 9 37 45 46 38 9 38 46 47 39 9 39 47 48 40 9 40 48 49 41 9 41 49 50 42 9 42 50 51 43 9 43 51 26 27 9 5 6 52 44 9 44 52 53 45 9 45 53 54 46 9 46 54 55 47 9 47 55 56 48 9 48 56 57 49 9 49 57 58 50 9 50 58 59 51 9 51 59 25 26 9 6 7 60 52 9 52 60 61 53 9 53 61 62 54 9 54 62 63 55 9 55 63 64 56 9 56 64 65 57 9 57 65 66 58 9 58 66 67 59 9 59 67 24 25 9 7 8 68 60 9 60 68 69 61 9 61 69 70 62 9 62 70 71 63 9 63 71 72 64 9 64 72 73 65 9 65 73 74 66 9 66 74 75 67 9 67 75 23 24 9 8 9 76 68 9 68 76 77 69 9 69 77 78 70 9 70 78 79 71 9 71 79 80 72 9 72 80 81 73 9 73 81 82 74 9 74 82 83 75 9 75 83 22 23 9 9 10 84 76 9 76 84 85 77 9 77 85 86 78 9 78 86 87 79 9 79 87 88 80 9 80 88 89 81 9 81 89 90 82 9 82 90 91 83 9 83 91 21 22 9 10 11 92 84 9 84 92 93 85 9 85 93 94 86 9 86 94 95 87 9 87 95 96 88 9 88 96 97 89 9 89 97 98 90 9 90 98 99 91 9 91 99 20 21 9 11 1 28 92 9 92 28 29 93 9 93 29 30 94 9 94 30 31 95 9 95 31 32 96 9 96 32 33 97 9 97 33 34 98 9 98 34 35 99 9 99 35 3 20

---

Could this be a possible reason for the BCs not quite showing the expected values? I also followed an example suggesting that if using a single block, there is no need to specify the mapped_cells file/partition and one can directly use the original su2 file; when I tried this approach and directly use test1.su2 file, I am met with the following error and the run does not proceed:

---

geom.geometry_exception.GeometryException@../geom/grid/usgrid.d(1738): cannot find face in collection

../geom/grid/usgrid.d:1738 void geom.grid.usgrid.UnstructuredGrid.read_from_su2_file(immutable(char)[], bool, double, bool) [0x935129] ../geom/grid/usgrid.d:837 geom.grid.usgrid.UnstructuredGrid geom.grid.usgrid.UnstructuredGrid.__ctor(immutable(char)[], immutable(char)[], bool, double, bool, immutable(char)[]) [0x8b52f9] ../geom/luawrap/luausgrid.d:277 newUnstructuredGrid [0x8ab238] ??:? [0xe284ef] ??:? [0xe36dc1] ??:? [0xe28803] ??:? [0xe2787f] ??:? [0xe28b32] ??:? lua_pcallk [0xe25483]

---

I have attached a .zip of my sample case with the included grid, output and run files for the former testing with partitioning. The cs.lua is my input deck and I based the setup off of the hollow-cylinder-flare (HCF) example case and my slurm job script is called eilmer_run.sh. The grid file boxx.su22 is copied to test1.su2 before running the partitioning script create_grid.sh.

If you can see what I may be doing wrong with specifying my BCs/seeing what else must be changed to make this work, it would be appreciated as this seems to be a small issue, but I have not been able to sort out why this occurs. I will include some images from my run below showing the grid and and resultant fields indicating the issue (when compared to inflow variables supplied in cs.lua script)

2) I wanted to learn more about the options in the steady-state solver; when looking through the various examples, I see various options and usages of the SS solver, but I was not able to find a similar reference guide as to what the various options would be for this. Is there a command/location in the source code I can refer to find out about this?

3) One other command I saw in some cases was to use the command 'config.propagate_inflow_data=true'; what does this do/refer to? I would presume it may be for special inflow BCs if a profile/data was supplied and/or moving mesh simulations but I could not find any documentation on this if you may be able to clarify this point.

Thank you and apologies if this concern may have been addressed prior. I see a lot of potential for the code and wish to contribute to it if I am able to use it for my work.

box.zip

[plaad]

Here is the sample output:

Apologies for the formatting but the pipe symbol separates the two outputs of the diff command where the original grid is of longer length.

The field variables are initialized as specified but the inflow velocity should be 0 at the bottom boundary.

[plaad]

One question I had was do you know where the ghost cell information is stored/if there is a file to which where the BC information is written to? Is there an option to write this information within the config subdirectory?

As shown from the contour plots, I am using an under-refined grid near the wall and can see slight changes in the velocity and temperate solution when probing the field close to the boundaries but I would like to ensure my isothermal BC is set to 290.0/velocity is 0 at the bottom no-slip wall boundary. I would think it is working correctly, due to the gradients in the field, but want to confirm via inspection of the ghost cells if possible.

Currently it does not appear to be the case in the contour plot (unless it is because my grid is under-refined/CFL is too large). I was able to probe the bottom surface solution but this returns the first cell-center location from the wall BC, and not the exact ghost-cell layer/bottom boundary explicitly.

[uqngibbo]

Hi plaad,

From what I can see here, all of this looks like expected behaviour. Partitioning a block into a single block does rearrange the points in the file, but should end up producing the same physical grid. The error you found must be related to something else.

The reason your wall boundary is not approaching the expected value is because your grid is very underresolved. The flow is being set to zero velocity and 290K at the interface: If you try refining the grid you should see the values in the first cell off the wall approaching those numbers.

Unfortunately the steady-state solver is not super well documented in eilmer4. It does work well and has been used extensively internally, but it's not an officially supported feature with good documentation. We are currently in the process of releaseing eilmer v5 (also called 'lmr') which does have better explanations, but if you want to use v4 you'll have to teach yourself a bit.

Finally, propagate inflow boundary conditions is a setting for the, now depreciated, space marching capability that was brought over from Eilmer 3. It copies a profile boundary condition through an entire domain, but only really is useful for very specific space marching calculations.

Let me know if you have any specific questions about the steady-state solver and I will be happy to answer them.

[plaad]

Hi uqngibbo,

Thanks for getting back to me. I did try using a more refined grid and it does produce the correct BC behavior so using the under-resolved grid was my mistake - however, can one probe the ghost cells directly from the post-processing routine (as I was only able to obtain information from the first cell center using the extract-surface command, not the boundary value itself as mentioned)? I think one thing that would be good to know is can one post-process on the fly so I can output contours/surface data while the run is ongoing.

I can always probe the boundary in Tecplot/PV after the run is done but when I tried to run a post-processing command during the run, it prevented the output of my conventional post-processing output files for some reason.

For the SS solver, I was trying with the following options used in some of the examples:

` SteadyStateSolver{ use_preconditioner = true, precondition_matrix_type = "ilu", frozen_preconditioner_count = 100; start_preconditioning = 1,

use_scaling = true, use_complex_matvec_eval = true,

number_pre_steps = 10000, number_total_steps = 100000, stop_on_relative_global_residual = 1.0e-10,

-- Settings for FGMRES iterative solver max_outer_iterations = 10, max_restarts = 10,

-- Settings for start-up phase number_start_up_steps = 0, cfl0 =0.10, eta0 = 0.1, tau0 = 1.0, sigma0 = 1.0e-30, p0 = 1.0,

-- Settings for inexact Newton phase cfl1 = 100.0, tau1 = 1.0, sigma1 = 1.0e-30, p1 = 1.0, eta1 = 0.1, eta1_min = 0.1, eta_strategy = "geometric",

-- Settings control write-out snapshots_count = 50, number_total_snapshots = 5, write_diagnostics_count = 20 }

`

What I had noticed is that my runs seemed to only proceed and stop after the 100 iterations. It would not proceed with the other outer iterations so perhaps I am missing something/have conflicting settings. Further, can you explain what eta, tau, sigma, p0, and eta variables correspond to? Apologies if I missed anything which details this from the online documentation, but I believe I could not find many details on this and tried my best to set this up based on the example in v4.

Thank you for your taking the time to clarify and help with these concerns; I sincerely appreciate it.

[uqngibbo]

Hi plaad,

To get data on the boundaries of a flow, you can use our "loads" files. An example of their use is in the repo in: gdtk/examples/eilmer/2D/flat-plate-transitional-sabcm

Briefly, to request loads be written you have to set the a label for the boundary you care about. This is the groups argument to the boundary condition:

north=WallBC_NoSlip_FixedT:new{Twall=300.0, group="wall"}

You then set the following arguments:

config.boundary_groups_for_loads = "wall"
config.write_loads = true
config.dt_loads=XXX

Where dt_loads is the frequency that the forces/flow on the boundary are written. These end up being plain text files in the loads directory. You should be able to plot them with a spreadsheet or python code.

For your steady-state calculations, the best thing is to use the fully preconditioned GMRES solver as it is set up in the examples. For steady-state simulations I would use the following:

config.with_local_time_stepping=true
config.extrema_clipping=false

SteadyStateSolver{
   use_preconditioner = true,
   precondition_matrix_type = "ilu",
   frozen_preconditioner_count = 25,

   use_scaling = true,
   use_complex_matvec_eval = true,
   use_physicality_check = true,

   number_total_steps = 1000,
   stop_on_relative_global_residual = 1.0e-8,

   -- Settings for FGMRES iterative solver
   max_outer_iterations = 40,
   max_restarts = 0,
   residual_based_cfl_scheduling = true,
   cfl_max = 1e4,

   -- Settings for start-up phase
   number_start_up_steps = 0,
   cfl0 = 5e-1,
   tau0 = 0.01,
   eta0 = 0.01,
   sigma0 = 1.0e-30,
   p0 = 0.5,

   -- Settings for inexact Newton phase
   cfl1 = 1e0,
   tau1 = 0.1,
   eta1 = 0.01,
   sigma1 = 1.0e-30,
   p1 = 0.8,
   eta_strategy = "constant",

   -- Settings control write-out
   snapshots_count = 100,
   number_total_snapshots = 2,
   write_diagnostics_count = 1,
   write_loads_count = 1000,
}

You may have to change number_total_steps for your application, that stops the solver at a given step count whether it's converged or not. You might also have to change cfl1. That is the starting cfl value, which might be a bit too high in some difficult cases. tau1 and p1 control the automatic cfl growth. a tau1 of 0.1 means the CFL will not grow until the global residual has dropped by an order of magnitude. If your cfl is growing too fast too early, you can change it to 0.01 to delay the growth. p1 is an exponent in the growth rate. If you want the cfl to grow faster, you can make it bigger, but stick to about 0.8-1.2 I would say.

You may also want to try using number_start_up_steps. This lets you do a number of steps with first-order spatial accuracy, in case you have a really difficult flow that won't start second order. cfl0, tau0, and p0 are corresponding grow settings for the startup phase, which are only used if number_start_up_steps is greater than 0.

Most of the other settings you can leave alone, I think.

Nick,

[plaad]

Ok, I actually had those options included in my script but commented them out since I wasn't sure if this was treated differently from the ghost cell layers. I didn't realize it could be used to probe other variables so thank you for pointing this out.

Also, understood with regards to the SS solver setup. Thanks for clarifying the various options - this helps a lot and gives me a sufficient understanding on what to adjust, if necessary.

A final question: I tried prior to post-process during an ongoing run on the fly but it created a .pvtu with only the initial conditions; the same ongoing run also had a post-processing step included in the job submission which would conventionally output a solution file for all of the time indices (tindx=all) but my intermediate output attempt made it so this output never occurred when the run finished.

Being said, is there a correct way to produce a solution file on the fly? I understand I can include the --report-residuals flag on the run command to probe solution progress as well. Apologies if I missed something, but the only thing I could guess would be if I output a snapshot of my transient/SS run which could be used to create a .pvtu file but I was not able to find an example for this or related details in the current docs.

Thanks for all of your help and time in answering these concerns, I should be good to go now otherwise.

[uqngibbo]

We don't often post process while the code is running, so I can't guarantee it will work perfectly, but in principle there's no reason why you can't run the post processor in one terminal while the other one is running a sim. For steady state simulations we often just look at the e4-nk.diagnostics.dat file. If that's something you are interested in I can send you a script to visualise it.

Nick,

[plaad]

Ok, maybe it is an issue on my HPC system/compute node as when I tried submitting a secondary post process script, it did mess up the original post-processing routine. That is, I submitted a job with the following commands in a slurm script:

prep-gas ideal-air.inp ideal-air-gas-model.lua e4shared --prep --job=cs mpirun -np 36 e4mpi --run --job=cs e4shared --post --job=cs --tindx-plot=all --vtk-xml --add-vars="mach,pitot,total-p,total-h"

but ran another post processing script during the job run time to try and generate a .pvtu output file with the command:

e4shared --post --job=cs --tindx-plot=last--vtk-xml --add-vars="mach,pitot,total-p,total-h"

And this produced an output file, but when the run eventually completed, the original post routine to plot all output instances never occurred despite the run proceeding to completion so maybe this can be something to try out in v4 (unless I used the wrong commands)? I think if you do not mind sharing that script, it would be helpful as I can send you an email for correspondence (if necessary) for this.

I actually saw in v5 docs today that you have something that is pretty much what I was looking for: "snapshot2vtk"; since snapshots would be produced for SS/transient runs at set intervals, this would allow output of the field on the fly so maybe I will have to wait until the next version may be released. Actually, it seems it nearly has most of the features I was thinking about.

As I have been using the code, one thing I recall not being too sure about was rotational periodic BCs: for structured grids, I follow the example for the periodic shear layer for conventional periodicity, but I typically work with SU2 imported, multiblock grids. From the docs, could I incorporate translational and rotational periodic BCs by using the ExchangeBC_MappedCell:new{transform_position, c0, n, alpha, delta, list_mapped_cells, reorient_vector_quantities, Rmatrix} command without and with some rotational axis specified, respectively?

Is there a more specific way to make an unstructured variant of the following command for use in the lua script that I may be missing?

for j=1, njb do connectBlocks(fba.blockArray[1][j], "west", fba.blockArray[nib][j], "east") end

Moreover, since I use the this BC type for the interior definition for the inter-block to block BC definition when using METIS partitioning, would there be any conflict in using this for the periodic BCs? I would think not since I would have differing definitions, but I thought to ask to confirm.

Further, if I add additional features which may be of use to the community, may I pass them along to you all to potentially include/consider for future releases? I understand one can fork the repo and pull/push but I wasn't sure of the exact protocol you all may work with if you allow users to provide such features.

Thanks again for your time and for answering all of my concerns Nick. Whenever you get a chance to get back to me, we can close this thread as I am now good to go.

-PAL

[uqngibbo]

From what I can see with that postprocessing it should have worked. Perhaps your --tindx-plot=all failed for some reason. Also, lmr snapshot2vtk is really just a renamed command that does the same thing, but you're very welcome to move to Eilmer 5 if you want.

As for the periodic boundary conditions, unfortunately I'm not aware of anyone who has used them in unstructured mode, and they may work, or may not. We do occasionally take pull requests if they are well made and come with a good test of the new functionality, so if you want to try and implement this yourself I'd be happy to stay in touch.

Nick

[plaad]

Got it, do you know how I can update to v5? When I tried to git clone it seems to pull the v4 version. I saw I can pull the new revision but was not sure if there was a different command or way to specify this.

I think I will try to add this feature (periodic/rotationally-periodic BCs for unstructured grids) so I will for sure reach out if I can manage this and provide a suitable example or two to showcase this.

[uqngibbo]

The v5 version is called 'lmr', and is in the repository you already have, sitting beside v4. You should be able to go:

cd src/lmr
make FLAVOUR=fast install

Both MPI and steady-state are now built by default, so there's no need for WITH_MPI or WITH_NK if you were doing that before.

[plaad]

Got it, thank you.

Also, just a question from a development perspective: how difficult would it be to adjust the existing the stencils to add higher-order schemes? If I develop this to perform scale-resolving simulation, I would need to add higher-order spatial schemes for example.

I saw implementing other flux schemes seems straightforward via modification of the fluxcalc.d routine, but for ghost cell treatment and such, if I wanted to add additional flux limiters/modify the stencil for 3rd/4th order schemes, would this require much effort/extensive modification of the existing code structure (in your opinion/experience)?

Thanks again as I will go ahead with working on the periodic BCs and reach out once more if I am able to make a suitable working example for you all to implement/use. Thanks for all of your help Nick.

[uqngibbo]

We already have a 4th order low dissipation scheme for scale resolving work, it's called 'asf' or alpha split flux.

You can make use of flux_calculator="adaptive_ausmdv_asf" for a blended flux that uses asf in smooth regions of the flow and ausmdv near shocks, but be warned that the vanilla shock detector is not really well suited for this. I've had to experiment with my own discontinuity detectors to get it to work well. Let me know if you want more info about that.

[plaad]

Ok good to know regarding this scheme. I just a quick question regarding the v5 install:

I ran the commands you mentioned given I have a working v4 build, but I am met with the following error during the make install process: ... inyendian.d \ -L-ldl -L-Wl,-E ./fluidfvcell.d(42): Error: variable lmr.fluidfvcell.savedGasState - default construction is disabled for type GasState make: *** [makefile:327: lmr] Error 1

In the makefile, I have not changed anything except enabling the same flags I had used for my v4 installation. I typically load modules for any required dependencies and have the same environment variables set up correctly for v5. In the makefile at this line, the code looks like:

$(DMD) $(FLAVOUR_FLAGS) $(DFLAGS) $(OF)$@ \ $(DVERSION)newton_krylov \ main.d \ lmrconfig_with_str_subst.d \

Do you know what this error could suggest?

[uqngibbo]

Try updating your version of the code with git pull, and check you have the latest LDC compiler. I have version 1.35, which seems to work.

[plaad]

Oddly enough, I get the same error after doing so; my LDC version is 1.39 so I should be ok and v4 still compiles fine after updating the code. Maybe there's some conflict with my modules or something on my hpc setting:

For reference the flags I am using in v4 are: DMD ?= ldc2 NVCC ?= nvcc GPP ?= g++ FLAVOUR ?= debug

# PLATFORM options are linux, macosx ifeq ($(shell uname -s), Darwin) PLATFORM := macosx else PLATFORM := linux endif $(info PLATFORM=$(PLATFORM))

WITH_MPI ?= 0 WITH_CHT ?= 0 WITH_NK ?= 0 WITH_PIR ?= 0 WITH_SSC ?= 0 WITH_OPENCL_GPU_CHEM ?= 0 WITH_CUDA_GPU_CHEM ?= 0 DEBUG_CHEM ?= 0 WITH_COMPLEX_NUMBERS ?= 0 WITH_FPE ?= 1 WITH_DIAGNOSTICS ?= 0 MULTI_SPECIES_GAS ?= 1 MULTI_T_GAS ?= 1 MHD ?= 1 TURBULENCE ?= 1 WITH_E4DEBUG ?= 0

Then for v5 I am using: MD ?= ldc2 NVCC ?= nvcc GPP ?= g++ # default is NOT to opt out of MPI build OPT_OUT_MPI ?= 0 MPI_IMPLEMENTATION := OpenMPI # default is NOT to use MPICH WITH_MPICH ?= 0

FLAVOUR ?= fast WITH_FPE ?= 1

# PLATFORM options are linux, macosx ifeq ($(shell uname -s), Darwin) PLATFORM := macosx else PLATFORM := linux endif $(info PLATFORM=$(PLATFORM))

WITH_OPENCL_GPU_CHEM ?= 0 WITH_CUDA_GPU_CHEM ?= 0 DEBUG_CHEM ?= 1 WITH_DIAGNOSTICS ?= 1 MULTI_SPECIES_GAS ?= 1 MULTI_T_GAS ?= 1 MHD ?= 1 TURBULENCE ?= 1 WITH_E4DEBUG ?= 1 WITH_CHECK_JAC ?=0

The rest of the makefiles were left as is with the comments (omitted here).

I am not sure what may be wrong but will keep trying; perhaps version 1.39 is not ideal for the D compiler

[plaad]

Hi Nick, just an update:

I believe I was able to compile and install v5 correctly (issue was that I was compiling with debug chem flag enabled and it was searching for an option that may have been missing from my v4 options that I am using). Being said my output looks like this from compiling and building with the debug/fast profile:

cp lmr gdtk-module /home/partth/gdtkinst/bin cp lmr-run lmrZ-run lmr-mpi-run lmrZ-mpi-run /home/partth/gdtkinst/bin cp lmr-verify /home/partth/gdtkinst/bin cp lua-modules/*.lua /home/partth/gdtkinst/lib/ cp lmr.cfg /home/partth/gdtkinst/etc/ cp ../../extern/lua-5.4.3/install/bin/* ../../build/bin cp ../../extern/lua-5.4.3/install/lib/lpeg.so ../../build/lib cp -r ../lib/* ../../build/lib cp ../lib/nelmin.lua ../lib/secant.lua ../../build/lib cp gdtk-module /home/partth/gdtkinst/share cp share/* /home/partth/gdtkinst/share cp -r ../../build/* /home/partth/gdtkinst

In the dir I see all of the executables and the process looks as if it has completed successfully. But when I tried to follow the v5 tutorial to ensure the executable was built correctly, it appears it has not as I get the following:

[partth@nova18-wide-41 lmr]$ lmr version bash: lmr: command not found

Further, I think there should be more object files if I had installed it correctly (see image):

Should this be the case? Since my paths for v4 are working I think everything else should be good. Please let me know when you may get a chance. Thanks Nick!

[uqngibbo]

Hi plaad,

This is good progress. I'm actually not sure why your command is not being found. The install process looks like it has copied the executables into /home/partth/gdtkinst, and if your .bashrc is being set to point there, then they should be found.

Can you double check your .bashrc has been set correctly? It may also help to look in gdtkinst to make sure nothing has gone awry.

[plaad]

I found the issue was that despite my .bashrc options:

# export path for D compiler for building Eilmer CFD code: export PATH=${PATH}:/work/sidgs/partth/codes/eilmer/dcomp/bin

# set environment for eilmer: export DGD_REPO=/work/sidgs/partth/codes/eilmer/gdtk # path to cloned github repr/build dir export DGD=/work/sidgs/partth/codes/eilmer/install # set install dir path export PATH=${PATH}:$DGD/bin # path to exec export DGD_LUA_PATH=${DGD}/lib/?.lua export DGD_LUA_CPATH=${DGD}/lib/?.so

The code was installing to my home dir. To fix this, I manually fixed the installation directory by modifying the INSTALL_DIR flag to be set to $(DGD) and both the eilmer v4 and v5 were able to be installed properly. However, when I was trying to run an old case which was working in my prior v4 installation, I get some kind of odd message:

globalconfig.FlowSolverException@main_with_rev_string.d(503): /work/sidgs/partth/codes/eilmer/install/lib output.lua:151: bad argument #1 to 'for iterator' (table expected, got nil) ---------------- main_with_rev_string.d:503 _Dmain [0xdae935] object.Exception@../util/json_helper.d(30): Failed to parse JSON from file: config/odw.config Message is: Unexpected end of data. (Line 39:0)

Looking this up in the output.lua (line 151), it appears the error is here: for k,v in pairs(_solidModels) do f:write(string.format(' "%s": \n', k)) f:write(string.format('%s,', v:tojson())) end

However, I am not using any solid mechanics coupling in this case (see input deck test.txt here for reference); I use the same makefile options as before which worked prior, but I modified the install dir manually:

# The install destination. #INSTALL_DIR ?= $(HOME)/gdtkinst # OLD option INSTALL_DIR ?= $(DGD)

Inspection of the output shows a problem with my build_config_files() routine:

Eilmer 4.0 compressible-flow simulation code. Revision-id: 76e87cbe Revision-date: Wed Aug 14 17:08:21 2024 +1000 Compiler-name: ldc2 Build-date: Tue Aug 27 09:58:00 AM CDT 2024 Build-flavour: debug Profiling: omitted Capabilities: multi-species-gas multi-temperature-gas MHD turbulence. Parallelism: Shared-memory Begin preparation stage for a simulation. Loading prep.lua... Done loading prep.lua The field 'title' cannot be set in 'config' table. nil GasModel has nsp= 2 nmodes= 0 Importing: su2-grid/block_0_test1.su2 ... (done) scale=1 dim=2 ncells=11516 nbfs= 4 Importing: su2-grid/block_1_test1.su2 ... (done) scale=1 dim=2 ncells=11494 nbfs= 4 Importing: su2-grid/block_2_test1.su2 ... (done) scale=1 dim=2 ncells=11513 nbfs= 3 Build config files for job: odw There was a problem in the Eilmer build function build_config_files() in prep.lua

I am following the oblique detonation example directly so there should be no issue with the title given that I ran this and other such cases fine before when the code was being installed to the home dir. Maybe I needed to set a global --prefix= when calling the make command to update all paths instead of just fixing the INSTALL_DIR manually?

test.txt

[uqngibbo]

I suspect this is a problem with having eilmer 4 and eilmer 5 installed in the same place. The two codes have slightly different lua backends with the same name, so they're clashing. I will talk to Rowan about this at our meeting this week.

In the meantime, it's probably best to remove your gdtkinst directory, and reinstall with either eilmer 4 or eilmer 5 depending on your use case.

Nick,

[plaad]

Yes, that was the case; I removed the install and just tried one install and it works. Sorry for the trouble and thanks for the help. I will try to move onto v5 now that it is working fine. Thank you Nick!