Formula to derive HPCG problem size - that fits in system memory

[puneet336]

As far as i know, Currently we need to experiment with different problem sizes NX,NY,NZ to find a problem size that can fit in memory. Is there a formula to derive a problem size for HPCG that consumes x% of memory theoretically?

I was looking for something which we already have for HPL, where say if a system has 1GB memory then system_memory=1G system_memory_bytes=1 1024 1024 * 1024 dp_elements = system_memory_bytes/8 PSize=sqrt(dp_elements)

and for using 80% memory size - PSize = PSize * 0.8

and further minor adjustment can be made to make PSize as multiple of problem size (NB)

although actual usage may vary based on various factors such as how many MPI ranks have been launched etc.

[maherou]

Here is a rough formula for computing the memory usage:

You can get a rough estimate of data usage by knowing that the finest grid matrix and related vectors use approximately for each MPI process:

27N - doubles, local ints, global ints (nonzero data for the sparse matrix) N - size_t (pointer data for the sparse matrix) 7N - doubles (vectors used in the solution process)

Where N is the product of the local grid dimensions, specified as input to the hpcg executable.

The size of local and global ints are compile-time parameters, but they are commonly 4 bytes (int) and 8 bytes (long long), so we will assume that size.

Then the sum is

27( 8 + 4 + 8 )N + 8N + 78N

Or 604N bytes per MPI process.

The coarse grids use a small fraction of the memory of the finest grid (the reduction is a factor of 8 per level), so let's say the total memory use by data objects is roughly 700N total bytes.

I hope this helps.