search

firemodels / fds

Fire Dynamics Simulator
https://pages.nist.gov/fds-smv/
Other
673 stars 627 forks source link

Development of optional pressure solver ScaRC #1105

Closed gforney closed 6 years ago

gforney commented 9 years ago 
Hi, Kevin and Randy,

as discussed during your visit in Berlin I just open a new issue for all questions
and comments related to the further development of the optional pressure solver ScaRC.

A first issue is based on the question how to treat the additional MPI-calls which
are needed in ScaRC. At the moment I sample all those exchanges (global scalar-products,
global l2-norms, own matrix-vector-multiplication, ...) in my own routine which requires
to include 'mpif.h' into ScaRC. This caused some compilations problems, such that we
can either move those routines to main.f90 where all the other communication routines
are located or have a closer look to a proper compilation. I would prefer the second
way because it allows for a cleaner differentiation of the several communication routines
and doesn't interfere with the original code. Please, let me know how to proceed.

Furthermore, I would like to ask it there is a way in the original communication routines
to additionally exchange the ghost cells. If I am right, then currently only, the cells
1 to KBAR are exchanged if one thinks i.e. of an exchange at IOR=-1 in a 2D-domain.
Is there also a way to simply extend the extisting communication routines for the exchange
of 0 to KBAR+1? As I suppose your I_MIN, I_MAX,...,K_MAX-arrays are only related to
a range from 1 to IBAR, .., KBAR.

Thank you,

Susan

Original issue reported on code.google.com by kiliansusan on 2010-06-21 11:33:21

gforney commented 9 years ago 
Will you include PRES_METHOD to the PRES-block or should I do that ??

Original issue reported on code.google.com by kiliansusan on 2010-07-28 13:23:22

gforney commented 9 years ago 
Susan,

I think Jason told me it is better to use integer comparisons (instead of character)
in SELECT and IF blocks.

R

Original issue reported on code.google.com by randy.mcdermott on 2010-07-28 13:30:44

gforney commented 9 years ago 
I do not want PRES_METHOD to be an input parameter. Whatever you currently use to indicate
which SCARC method to use, let this indicate that PRES_METHOD='SCARC'. We usually let
the fact that a user specifies a certain parameter to indicate that the user wants
to use a certain feature. The less number of input parameters, the better.

Original issue reported on code.google.com by mcgratta on 2010-07-28 13:31:42

gforney commented 9 years ago 
If the SELECT CASE statements are evaluated many times, then it is better to use integers.
However, if we just execute once in a routine, it does not matter.

Original issue reported on code.google.com by mcgratta on 2010-07-28 13:33:59

gforney commented 9 years ago 
So, do I understand it correctly, that PRES_METHOD='FFT' by default (and we use the
character comparison further on)? So this is just declared in cons.f90, isn't it ?

And we don't specifiy PRES_METHOD as user input? The only way to change its value is
to set e.g. SCARC_METHOD='CG' in READ_PRES and this automatically leads to PRES_METHOD='SCARC'
?

Original issue reported on code.google.com by kiliansusan on 2010-07-28 13:48:34

gforney commented 9 years ago 
Yes, there are a few pressure-related constants in cons.f90. Check the code to make
sure that SCARC_METHOD is working properly in all places.

Original issue reported on code.google.com by mcgratta on 2010-07-28 14:22:28

gforney commented 9 years ago 
in cons.f90:

! Variables related to pressure solver

LOGICAL :: ITERATE_PRESSURE=.FALSE.,PRESSIT_ACCELERATOR=.FALSE.
...
INTEGER :: PRESSURE_ITERATIONS=0,MAX_PRESSURE_ITERATIONS=10000
CHARACTER(10) :: PRES_METHOD='FFT'

at the end of READ_PRES:

IF (SCARC_METHOD /= 'null')  PRES_METHOD = 'SCARC'

Ok ?

Original issue reported on code.google.com by kiliansusan on 2010-07-28 15:06:21

gforney commented 9 years ago 
Yes, perfect.

Original issue reported on code.google.com by mcgratta on 2010-07-28 15:20:05

gforney commented 9 years ago 
I commited a new version of ScaRC which already includes the PRES_METHOD variable. As
already discussed, this variable is set to 'FFT' by default and only changed to 'SCARC'
if SCARC_METHOD is set in the &PRES-block. Only if PRES_METHOD=='SCARC' the whole ScaRC-initialization
is done in main. 

In velo.f90 and divg.f90 there are two if-statements related to PRES_METHOD now. These
lines of code are only performed if PRES_METHOD /= 'SCARC', that is, only for the FFT-solver.
I indicated those IF-statements with 'NO_SCARC_IF'. Please feel free to change it if
you don't like it in this way. We could also use 'FFT_IF' instead or simply nothing
...

Finally, the call of the pressure solver depends on PRES_METHOD now, please have also
a look to this new structure.

Note, that for the examples in scarc2d of Comment 40 same results of serial and parallel
simulations can only be obtained if M%DT has the same initial value for both.

Original issue reported on code.google.com by kiliansusan on 2010-07-30 13:40:12

gforney commented 9 years ago 
Just for your information: I succeeded to get first versions of ScaRC-CG and ScaRC-BICGstab
to run in the 1-mesh and in the multi-mesh case in 3D !

The ScaRC-CG variant already ran before in 3D, but now it is based on my new data structures
which I introduced in order to handle the hierarchy of different grid resolutions for
the MG-version. These MG-structures are now the base of the whole ScaRC algorithm.
So, if only the finest grid level is used - as for CG - than I simply  treat this as
special case of the MG-structure.

To this end, I expanded my favorite test geometry to the 3D-case, see attachment. For
this geometry I tested different subdivisions in x-, y- and z-direction. The  CG- and
BICGstab-variants for the different multi-mesh-cases seem to be completely consistent
to the corresponding 1-mesh-case  (provided that I use the same initial M%DT for all
different subdivisions).

Even the  ScaRC-MG variant for the 1-mesh-case already runs. 

All serial variants (CG, BICGstab and MG)  produce exactly the same solution as the
original FFT-variant for the 1-mesh-case which can be seen be a smokediff comparison,
see attachment. 

For the parallel MG-variant there is still a little work to be done concerning the
communication structures on the coarser grid levels, but I am very confident to get
it to run shortly. Afterwards, I'll begin with some extensive tests, also with respect
to different grid resolutions on neighboring meshes and cases where e.g. two meshes
meet one ...

Original issue reported on code.google.com by kiliansusan on 2010-08-27 11:42:43

gforney commented 9 years ago 
(No text was entered with this change)

Original issue reported on code.google.com by kiliansusan on 2010-08-27 11:44:23

gforney commented 9 years ago 
What do you use for a convergence tolerance with the MG case? To get this perfect match
in velocity, don't you have to iterate the residuals to a very small value? Or maybe
I am misinterpreting what you mean by MG.

Original issue reported on code.google.com by mcgratta on 2010-08-27 11:50:41

gforney commented 9 years ago 
Kevin, I'll answer your question in about 2 hours, have to go to a meeting now ... Till
soon

Original issue reported on code.google.com by kiliansusan on 2010-08-27 11:53:58

gforney commented 9 years ago 
Just to avoid linguistic misunderstandings: If I talk about 'complete consistency of
1-mesh and multi-mesh-cases', I mean that the multi-mesh solution is exactly the same
as the 1-mesh solution (up to rounding error accuracy), only distributed over the single
meshes. Indeed, this can only be achieved if I use Jacobi-preconditioning because the
block-Jacobi preconditioner (multi-mesh case) is exactly the same than the global Jacobi-preconditioner
(1-mesh case).

When I compare the 1-mesh ScaRC-variants with the 1-mesh FFT I used  a convergence
tolerance of 1.0-16 before. Otherwise I wouldn't compute the exact solution and couldn't
compare with the FFT solution. But I just did some tests now: Even if I only take 1.0-10
tolerance for this special example I get a smokediff-error of 1.0-14 !  Taking 1.0-9
as convergence tolerance there are smokediff-errors in the range of 1.0-10 compared
to the exact FFT solution. If I take only 1.0-6 I get smokediff errors in the range
of 1.0-7, please see the attached pictures (tolerance is indicated at the end of the
picture names).

At the moment I cannot speak about the multi-mesh ScaRC-MG because there are still
a few index implementations on the coarsest grid level necessary to do the coarse grid
problem in parallel correctly (as told before there is not much to do to get it to
work). What is working at the moment is the multi-mesh ScaRC-CG and multi-mesh ScaRC-BICGstab.
But ScaRC-MG works fine for 1-mesh cases, that's the first important step towards the
multi-mesh version.

But generally: I always have perfect velocity matchings along internal boundaries.
If I iterate up to 1.0-16 then I get the exact solution just as the FFT. If I don't
iterate up to rounding error accuracy then I won't surely  have the exact solution,
but a solution which is very near to it. It may depend on the special problem what
tolerance must be used. But in every case (up to rounding error or only 1.0-6) I will
always have consistency of all computed quantities along internal boundaries, such
as velocity, just by construction of the method. The multi-mesh case does the same
matrix-vector multiplication than the 1-mesh case would do, the method is defined globally,
only performed locally in a data-parallel way.

As far as I know, the current smokediff-version only works for 1-mesh cases, is that
right? What about a multi-mesh version of it such that a 1-mesh computation could be
compared with a corresponding multi-mesh computation? Would it be a lot of work to
have such a version ? This would really be great for me (@Glenn: Please forgive me
this proposal if it would cause a huge amount of work ...)

Concerning the measurement of the velocity errors: I think it would be helpful if I
just measured the velocity errors to point out that there are none. Could you please
tell me again what I have to use in the fds-file to do just one pressure iteration,
namely one ScaRC-cycle, and than measure the velocity error without doing the pressure
iteration ?

Original issue reported on code.google.com by kiliansusan on 2010-08-27 14:40:20

gforney commented 9 years ago 
Just an idea: Probably I could use your coarse grid matrix A and the GAUSSJ routine
from CORRECT_PRESSURE to solve the coarse grid problem within the MG-variant. I'll
have a look at it ...

Original issue reported on code.google.com by kiliansusan on 2010-08-27 14:58:37

gforney commented 9 years ago 
Set VELOCITY_TOLERANCE to a value that is relatively large and that should force you
to go through one pressure iteration with a check of the velocity error.

Original issue reported on code.google.com by mcgratta on 2010-08-27 15:28:32

gforney commented 9 years ago 
Kevin, am I right that the measurement of the VELOCITY_ERROR not only indicates differences
between neighboring meshes but also e.g. in case of the obstruction at the bottom of
our example even in the 1-mesh case (in this special case differences between the given
inflow velocity and the computed ones? 

I am just wondering about different values for W_NEW and W_NEW_OTHER in the middle
of the obstruction which I got for the 2-mesh case. 

The computation of W_NEW_OTHER is based on UWP which is done in divg.f90 which includes
the inflow velocity of -0.5 as indicated in the geometry file. 

But the computation of W_NEW gets a different value than W_NEW_OTHER. The computation
of W_NEW includes the difference of the H-values in z-direction, so this is grid depended
while the computation of W_NEW_OTHER is not (only depends on the inflow velocities
and the current time). Is that all correct? 

Even if I run  COMPUTE_VELOCITY_ERROR for the 1-mesh case I get an error here. I am
confused because I thought the measurement of VELOCITY_ERROR is only related to multi-mesh
cases.

This may be a misunderstanding of me related to the VELOCITY_ERROR-routine. Can you
please help me to get rid of this confusion?

Original issue reported on code.google.com by kiliansusan on 2010-08-27 18:23:36

gforney commented 9 years ago 
VELOCITY_ERROR also includes the difference in the normal velocity at solid, internal
obstructions. The FFT pressure guarantees machine accuracy at the exterior boundary,
but not at the internal solid boundaries.

Original issue reported on code.google.com by mcgratta on 2010-08-27 18:27:49

gforney commented 9 years ago 
So I must not be confused if I don't have a non-zero velocity-error at the obstruction
in the multi-mesh case ? I even don't have it in the 1-mesh case. 

I was wondering because I really had the same values for H, U, V, W, FVX, FVY, FVZ
in the 2-mesh case as in the 1-mesh case up to rounding error tolerance. Nevertheless
there was this velocity-error ...

Thank you for this explanation ...

Original issue reported on code.google.com by kiliansusan on 2010-08-27 18:36:07

gforney commented 9 years ago 
Just for your information: It seems that the multi-mesh ScaRC-MG runs in 3D now! 

I just tested different subdivisions of my 3D test geometry. It seems to work fine
for different numbers of grid refinement levels (not only the 2-level variant, but
also more different refinement levels) and also  for the pointwise setting of boundary
conditions. 

To be really sure, I'll do more tests on some more complicated geometries. Besides,
I have to clean up the code again (many debug messages at the moment ...) and then
I'll do another commit.

I didn't check the 'many meshes meet one mesh'-variant for 3D-ScaRC-MG  yet. But the
related data structures are already included in the code and it should work. I will
surely check that carefully, too.

Original issue reported on code.google.com by kiliansusan on 2010-09-02 12:19:33

gforney commented 9 years ago 
Based on our recent discussions per mail I committed a small verification case. Please
see the attached directory for the resulting csv-files and the corresponding evaluation.
All test computations are done based on a new checkout from this morning.

Original issue reported on code.google.com by kiliansusan on 2010-09-10 11:46:32

gforney commented 9 years ago 
[mcgratta@fire70 Pressure_Solver]$ fds5 scarc3d_bicg_2meshx.fds
forrtl: No such file or directory
forrtl: severe (29): file not found, unit 51, file /home/fire52/mcgratta/FDS_Repository/Verification/Pressure_Solver/msg/001_scarc

Why do we need to look for this file?

Original issue reported on code.google.com by mcgratta on 2010-09-10 12:31:58

gforney commented 9 years ago 
Sorry, I forgot to set the debug level of ScaRC to zero. Here are the correct files
... I also commited the corrected FDS-files to the repository.

Original issue reported on code.google.com by kiliansusan on 2010-09-10 13:41:55

gforney commented 9 years ago 
mpirun -np 2 ~/FDS_Re*/FDS_C*/mpi_intel_linux_32_db/fds5_mpi_intel_linux_32_db scarc3d_bicg_2meshx.fds
Process   0 of   1 is running on fire70
Process   1 of   1 is running on fire71
Mesh   1 is assigned to Process   0
Mesh   2 is assigned to Process   1
forrtl: severe (193): Run-Time Check Failure. The variable 'scarc_solver_mp_scarc_exchange_$NM'
is being used without being defined

Run these cases with all debugging turned on. I fixed one problem with allocation of
TUSED_SCARC and committed this change. I do not want to fix this bug reported here
because I worry about breaking something. Be careful how data is used within modules.
Use the debugger to help you.

Original issue reported on code.google.com by mcgratta on 2010-09-10 14:30:34

gforney commented 9 years ago 
Ups, I don't have any idea what this is. I didn't have this problem, so I have to search
for it ...

Original issue reported on code.google.com by kiliansusan on 2010-09-10 14:54:56

gforney commented 9 years ago 
Could you please try again, I suppose I found the error. Unfortunately, I have to stop
in about 10 minutes for today because of a date. Please, let me know if it works now.

Original issue reported on code.google.com by kiliansusan on 2010-09-10 15:04:53

gforney commented 9 years ago 
This is about "scoping rules". You have defined NM locally within the SUBROUTINE SCARC_EXCHANGE,
but you have not given it a value or passed it as an argument. I assume that the value
is defined elsewhere in the module, but it is not clear to the compiler which value
of NM to use -- the one defined in some other subroutine or the one defined in SCARC_EXCHANGE.
We typically define the variables that are used within the subroutine locally, and
we pass in values needed from elsewhere. We try to avoid globally defining variables
at the top of the module unless these variables are needed throughout the module.

This is why I always run in debug mode when developing new code. If you run in release
mode, these minor mistakes are ignored, but you can end up with results that are not
expected.

Original issue reported on code.google.com by mcgratta on 2010-09-10 15:09:06

gforney commented 9 years ago 
Yes, you are right. This single definition of NM was a relic from an older version of
the code which I wasn't aware of. Indeed, I compiled it in debug mode, but this time
only with the gnu-compiler. And it didn't complain about it. I will have a 'global'
look to the routine if really all globally defined variables are really needed globally.
I think most variables are, but due to my recent restructurings  there might be some
new possibilities to get rid of some remains. Thanks for letting me know.

Original issue reported on code.google.com by kiliansusan on 2010-09-10 15:20:02

gforney commented 9 years ago 
It works now.

Original issue reported on code.google.com by mcgratta on 2010-09-10 15:29:00

gforney commented 9 years ago 
Be careful where you place DEVC's. You put them all at the corner of 8 cells, right
in the middle of the domain. For multi-mesh cases, I am not confident always that the
values are available. I rewrote your DEVC lines for
scarc3d_fft_1mesh.fds and scarc3d_bicg_2meshx.fds, and now the quantities agree.

&DEVC XYZ=0.31,0.39,0.39, QUANTITY='U-VELOCITY',ID='U-VEL'    /
&DEVC XYZ=0.31,0.39,0.39, QUANTITY='V-VELOCITY',ID='V-VEL'    /
&DEVC XYZ=0.31,0.39,0.39, QUANTITY='W-VELOCITY',ID='W-VEL'    /
&DEVC XYZ=0.31,0.39,0.39, QUANTITY='PRESSURE'  ,ID='PRESSURE' /
&DEVC XYZ=0.31,0.39,0.39, QUANTITY='H',ID='H'    /
&DEVC XYZ=0.31,0.39,0.39, QUANTITY='F_X',ID='F_X'    /
&DEVC XYZ=0.31,0.39,0.39, QUANTITY='F_Y',ID='F_Y'    /
&DEVC XYZ=0.31,0.39,0.39, QUANTITY='F_Z',ID='F_Z'    /

Original issue reported on code.google.com by mcgratta on 2010-09-10 19:14:39

gforney commented 9 years ago 
I changed the scarc3d...fds files to account for the DEVC placement. I also added a
case called scarc3d_fft_2meshx.fds. This case uses my iterative technique for the pressure
solver. Note that I changed the BC. Instead of the INFLOW coming in from an OBST, I
created a VENT at the boundary of the domain instead. Reason is that when fluid is
introduced at an interior boundary, it is not exact. My iterative pressure scheme forces
the INFLOW to be within a specified velocity tolerance. It does the same at interpolated
boundaries. I want to be able to compare the methods, so I made the change. All six
cases should now produce the same output. I don't think it is exactly the same, but
to the eye it is exact. My iterative scheme is slow, but it is possible to drive the
velocity components even closer than I have set. 

Will be gone Monday-Wednesday next week. Randy knows what I am doing, so you can ask
him questions.

Original issue reported on code.google.com by mcgratta on 2010-09-10 19:36:54

gforney commented 9 years ago 
Please not, that the JACOBI preconditioning is the slowest thing I can do. I just took
this preconditioner because it's the only one which produces the same results in its
blockwise version (its parallel, mesh-wise version) as the serial 1-mesh version. I
think I already mentioned that before. For all other preconditioners the blockwise
versions get slightly different results compared to the corresponding 1-mesh version.
This should not matter if one iterates till rounding error accuracy, but it will surely
matter if one only iterates till e.g. 1E-8 or something like that. 

It will be another important part of the verification stuff to analyze at what accuracy
the pressure solver must iterate. Up to now I don't think that we really have to iterate
till rounding error accuracy. My impression is that it is enough to iterate till 1E-10
or 1E-12. But, most probably it depends on the underlying problem. So, I will have
to examine that carefully. 

All in all, I would like to concentrate on the JACOBI preconditioner for consistency
checks like this because it enables me to compare the serial and parallel versions
point- and cellwise. But, please keep in mind that this is by far the less efficient
version of ScaRC with respect to computational speed! There will also be cases (more
complex geometries) where this preconditioning won't be strong enough. Once I am sure
that there is full consistency of the parts surrounding the pressure solver, I will
go over to multigrid or combinations like cg/bicgstab with multigrid preconditioning
which will be much more efficient. 

Besides I think that we should use a much finer grid resolution with much more grid
cells later during the test phase. I would like to test also subdivisions with a higher
number of meshes, not only the present 2x1x1, 1x2x1 and 1x1x2 cases with 2 meshes,
but also 8 submeshes (2x2x2), 27 meshes (3x3x3) and 64 meshes (4x4x4), just to have
one or more fully interior meshes. I'll come back to this issue later.

Concerning your explanations to the present cases I'll write a new thread at Monday.

Original issue reported on code.google.com by kiliansusan on 2010-09-11 06:05:31

gforney commented 9 years ago 
Indeed, the change of the BC's improves the situation a lot, that's great. But please
see the attached pdf-file for more remarks to this issue, there are still some differences
between serial an parallel left and I would appreciate it very much if you had a closer
look to it because it's very important for me.

At the moment I am running several multi-mesh computations (comparison of the original
FFT, the new FFT-iteration scheme, ScaRC-BICG and ScaRC-MG on 1-, 8- and 64-mesh subdivisions).
I will summarize my results as soon as possible.

Original issue reported on code.google.com by kiliansusan on 2010-09-13 15:00:15

gforney commented 9 years ago 
Just another comment: I decided to implement a FFT-preconditioner for the CG-, BICG-
and MG-solver. Maybe (most probably) this will be faster and surely more exact than
the the usual SSOR or GS-ADI-preconditioners. Due to my experience, the best preconditioning
was to use the GS-ADI before. That is an alternated backward elimination of extended
tri-diag-systems for all directions x, y and z (corresponding LU-decompositions are
done during the initialization phase). If you are interested, I'll explain more details
of this method. But at the moment, I hope that FFT will be better (and more suitable
to FDS).  To have a combination as global multigrid with blockwise FFT-smoothing would
certainly mean, that the local FFT's have to be performed more than once per time step.
But it's the same with the current pressure iteration scheme, where it also has to
be performed for many times. Probably, the stronger global coupling be the surrounding
multigrid will prevent too many local FFT solves. I'll check that.

As discussed per mail, let's chat about the rest ...

Original issue reported on code.google.com by kiliansusan on 2010-09-13 16:39:51

gforney commented 9 years ago 
Susan,

I am very glad to hear about the FFT preconditioner option.  This is something we have
had in mind all along.  Nice work.

R

Original issue reported on code.google.com by randy.mcdermott on 2010-09-13 18:08:23

gforney commented 9 years ago 
I committed code that hopefully improves the consistency of single and multiple mesh
calculations. It is not necessary to calculate special values of TAU_E and OM_E (stress
tensor and vorticity components) at interpolated boundaries. In VELOCITY_BC, we now
just update the appropriate values of U, V, and W in the cell that overlaps the other
mesh. Try the case again and let me know if it works.

Original issue reported on code.google.com by mcgratta on 2010-09-17 14:11:49

gforney commented 9 years ago 
Kevin, thank you very much!!  Indeed, I've played around with the code and omitted the
new settings of  TAU_E at internal boundaries which led to consistency for the considered
case. So, this seems just the change that you did and I'll check that again.

I ran some test cases meanwhile which I still have to summarize. Most probably, it's
best if I recompile the code with your newest changes and rerun those cases in order
to see if something changes. 

Another thing which I am wondering about is the value of MU at the beginning of VELOCITY_FLUX.
As far as I see MU isn't exchanged just before the call of this routine, is that true?
I had a case with changing values of MU where the ghost values on one mesh haven't
been consistent to the corresponding values of the neighboring meshes (see the position
where the vorticity and stress tensors are computed in VELOCITY_FLUX). For the values
of MUX, MUY and MUZ the ghost values of MU are used which have direct influence to
the definition of TXY, TXZ and TYZ which in turn have influence to the definitions
of FVX, FVY and FVZ. So, this might be another reason for inconsistencies.

Original issue reported on code.google.com by kiliansusan on 2010-09-17 15:04:47

gforney commented 9 years ago 
I performed a variety of test computations, please see verification_scarc4.pdf and the
corresponding test files in the attachments.

Original issue reported on code.google.com by kiliansusan on 2010-09-20 07:42:06

gforney commented 9 years ago 
Now, I debugged the inconsistencies again. For our simple test case I could find a work-around
for which it seems to work, please see verification_scarc5.pdf, the corresponding fds-files
and a picture for the pressure device. Hopefully, we can do this in general.

Susan

Original issue reported on code.google.com by kiliansusan on 2010-09-20 20:33:45

gforney commented 9 years ago 
Sorry -- I thought that I put in logic to stop the use of TAU_E and OM_E at INTERPOLATED
boundaries. There were still cases where these terms were being set. I no longer set
TAU_E or OM_E if EDGE_TYPE(IE,:) = INTERPOLATED_EDGE. The case now works as you have
it.

Original issue reported on code.google.com by mcgratta on 2010-09-20 21:59:11

gforney commented 9 years ago 
The devil is in the detail, I really know that. Thank you!  A short test for a coarse
8-mesh case also works completely consistent now. I'll restart the test cases from
comment 87 and come back to you with the results.

Original issue reported on code.google.com by kiliansusan on 2010-09-21 06:07:58

gforney commented 9 years ago 
I ran some test cases with the new version of velo.f90 based on our recent geometry
with subdivisions into 8, 64 and 256 meshes, please see the attached verification_scarc6.pdf
for the results.

Original issue reported on code.google.com by kiliansusan on 2010-09-23 14:28:57

gforney commented 9 years ago 
I'll check the FFT results. Commit these cases to the repository so that we can make
sure that we are comparing the same cases. Thanks.

Original issue reported on code.google.com by mcgratta on 2010-09-23 15:14:06

gforney commented 9 years ago 
I also tried to ran a 512-mesh case with a subdivision into 8x8x8 meshes. For this computation
I had 128 cores at my disposal. But, this computation always broke down with an error
message that there is too less memory. I was wondering about this because the local
number of grid cells is very small and 4 jobs should really fit into one core. 

Now, I tried to find out the reason for that and realized that the arrays I_MIN, ...
and all other arrays of shape (NMESHES,NMESHES) seem to be responsible for this huge
memory requirements. Indeed, in analogy to the existing structures I also use arrays
like this for my communication structure on coarser meshes. 

If the number of subdomains is small, then this concepts works great. But if there
is a very big number of subdomains then those arrays are very sparse, there are only
a few positions different from zero.  If the aim is to have applications with hundreds
of meshes, it could be advantageous to make this mechanism more sparse. What is your
opinion to that?

Original issue reported on code.google.com by kiliansusan on 2010-09-23 15:22:56

gforney commented 9 years ago 
It's a good idea to recheck that.  I'll commit the complete cases immediately tomorrow
morning (have to leave now because of date ...). As an attachment you'll already find
the FFT-cases.

Have a nice day, 
Susan

Original issue reported on code.google.com by kiliansusan on 2010-09-23 15:29:33

gforney commented 9 years ago 
(No text was entered with this change)

Original issue reported on code.google.com by kiliansusan on 2010-09-23 15:30:57

gforney commented 9 years ago 
I committed the current test cases. Please have a look that everything is really correct.

Original issue reported on code.google.com by kiliansusan on 2010-09-24 11:18:17

gforney commented 9 years ago 
Some remarks/questions:
1) As the next step I intend to implement the FFT smoother. I don't like the SSOR smoother
because it is relatively sensitive with respect to the method parameters, especially
the relaxation parameter. Due to my experience the GSADI-smoother is much less sensitive
in this regard, therefore I would like to implement it directly after the FFT-smoother.
GSADI would have the advantage that it also allows for variable coefficients in the
system matrix.
2) Do you agree that I print out some convergence info about the used ScaRC-solver
in dump.f90 within WRITE_DIAGNOSTICS (only if PRES_METHOD='SCARC' corresponding to
the additional print-outs for the iteration scheme) ?

Original issue reported on code.google.com by kiliansusan on 2010-09-24 12:03:15

gforney commented 9 years ago 
Last remark for today: If the test cases are too fine with respect to the grid resolution
and too big with respect to the number of meshes for a standard verification case,
we can omit the 256-mesh case and choose coarser grid sizes for the other ones. It
was only a test with respect to scalability ...

Have a nice weekend, Susan

Original issue reported on code.google.com by kiliansusan on 2010-09-24 14:28:32

gforney commented 9 years ago 
I added the 8 mesh fft case to the Repository. I set the VELOCITY_TOLERANCE to 0.01.
The results are OK, but not perfect. 

I suggest you add SCARC output to the .out file. Look at one of the fft cases with
VELOCITY_TOLERANCE set. Then search in dump.f90 for where that is written.

Original issue reported on code.google.com by mcgratta on 2010-09-24 20:49:18