Zoltan2: Possible scaling issue with MueLu/Z2multijagged for MueLu coarse level repartitioning?

pwxy commented 7 years ago

I observed the following scaling of "MueLu: Zoltan2Interface: Zoltan2 multijagged (sub, total, level=2)" (the time for Zoltan2::PartitioningProblem->solve()) on the LLNL IBM BG/Q platform for strong scaling for the Drekar Poisson test case. Started with a 2.4B row matrix, but Zoltan2 reparitioning not called until after two levels of MueLu aggregation (~700x factor reduction). So have the case with few rows of the matrix per MPI process (probably not the standard usage of Z2 in most apps):

MPI "MueLu: Zoltan2Interface: Zoltan2 multijagged (sub, total, level=2)" time in sec 131072 2.10 262144 524288 12.25 1048576 26.7 1572864 66.9

I built the muelu driver on solo and ran with 256, 512, 1024, 2048, 4096 and 8192 MPI processes and could see that the Zoltan2 multijagged isn't scaling as well as hoped (but it is definitely easier to see the problem at much larger scales).

This is strong scaling with "Matrix type: Brick3D" (27 nnz per row) with problem size of 81M rows. Zoltan2 is not called until after two levels of coarsening (each coarsening reduces the rows by factor of roughly 27), so for example the 1024 MPI case, the matrix Z2 gets is 118,000 rows.

Times are the max over MPI processes for "MueLu: Zoltan2Interface: Zoltan2 multijagged (sub, total, level=2)" (this is the time for Z2 MJ to construct the new partitioning; MueLu tells Z2 how many partitions are needed and MueLu migrates the data afterwards) for both "mj_migration_type"=0 and "mj_migration_type"=1 performed 3 runs of each and reported the lowest time below

MPI MJ=0 MJ=1 256 0.0060 0.0060 512 0.0091 0.0090 1024 0.0144 0.0142 2048 0.0247 0.0244 4096 0.0607 0.0605 8192 0.1091 0.1089

So unless I screwed up, there doesn't seem to be much difference between "mj_migration_type"=0 and "mj_migration_type"=1

On solo the only module change I made was "module swap intel intel/17.0.4.196"

cmake file attached; muelu xml file attached

Here are my input arguments to the muelu driver:

MueLu_Driver.exe --matrixType=Brick3D --nx=433 --ny=433 --nz=433 --mx=${xproc} --my=${yproc} --mz=${zproc} --xml="muelu_scaling.xml"

MPI xproc yproc zproc

256 8 8 4 512 8 8 8 1024 16 8 8 2048 16 16 8 4096 16 16 16 8192 32 16 16

cmake_muelu_kokkos_serial_serrano_icc17.txt muelu_scaling.xml-z2mj_mj0_lev2minpp1024-c1000-t_exp-remap_rebpr-1vcyc11.txt

aprokop commented 7 years ago

@trilinos/zoltan2 @trilinos/muelu

I wonder if rcb works well.

mndevec commented 7 years ago

@jjellio @pwxy @jhux2 @srajama1 @kddevin As far as I understood, MJ is called in Multigrid when the number of coordinates are significantly reduced. On the examples from Paul, MJ has very few coordinates per processors O(10), and the number of processors are as high as 8192.

Having such few coordinates per processor makes MJ communication bound, as a result it does not strong scale. Also, the processors are having difficulty to find the location of the cut when they have such a small local dataset.

My latest push introduces a new Zoltan2 parameter, mj_premigration_option.

       <Parameter name="mj_premigration_option" type="int" value="VALUE"/>

Let's say MJ is called with "n" processors to partition to "p" parts. "G" is the global number of coordinates:

Default of VALUE is 0, which means usual case.
If VALUE=1, 1- if n > p. The result parts are less than the number of processors --> this is the case of muelu. 2- if G/n < 10000, the average number of local coordinates per processor is less than 10000. ("10000" is hard coded for now, but I will have a parameter that defaults to 10000). Then, MJ will first gather the coordinates to "p" processors using modular arithmetic (referred as Premigration in timers), and do the partitioning using p processors, and redistribute it (referred as Postmigration in timers). If the VALUE is 1, then the consecutive ranks (0, 1, 2, ... , p -1) will be chosen to migrate coordinates to. The good thing about this is, the ranks (0 to p-1) are likely to be in a node, so the run of MJ will be localized without network access. However, this will cause premigration to gather all coordinates to single node, which may cause network bottlenecks.
To prevent such bottlenecks, VALUE can be set to any positive integer. Max value will be n / p, if given larger than this it will be automatically set to this number. Basically, the gather operation will be done to the processors with ranks: (0 x VALUE, 1 X VALUE, 2 x VALUE, ...., (p-1) x VALUE). So if VALUE is chosen as the number of ranks per node, the migration can be distributed to evenly to the nodes.

I ran small scale tests on bowman upto 512 ranks. This has improvements over the previous method. I am not sure how much it will help on O(100K) processor cases.

@jjellio , @pwxy :

Would that be possible to run the simulations again using:

    <ParameterList name="repartition: params">
       <Parameter name="algorithm" type="string" value="multijagged"/>
       <Parameter name="mj_premigration_option" type="int" value="1"/>
       <Parameter name="timer_type" type="string" value="macro_timers"/>
       ....
    </ParameterList>

and one more with value =16 or 64 -- the number of ranks per node.

    <ParameterList name="repartition: params">
       <Parameter name="algorithm" type="string" value="multijagged"/>
       <Parameter name="mj_premigration_option" type="int" value="16"/>
       <Parameter name="timer_type" type="string" value="macro_timers"/>
       ....
    </ParameterList>

I hope this will solve the problem.

@aprokop : I ran RCB on small scale instead of MJ, its times were about the ~twice of MJ (64 - 512 cores). I am not sure how it will scale on larger scales, but I think it will run into the same problem.

jjellio commented 7 years ago

I'll work on getting this tested on Cori.

aprokop commented 7 years ago

@mndevec First of all, thank you very much for working on this.

I would like to add some comments to @mndevec's description.

It is correct that in our case n > p always. We have a knob that controls how p is calculated. This is done through a "repartition: min rows per proc", which is by default set to 2000. So, if we want to repartition n nodes, we set p ~= N/2000. However, the important thing to notice that the number N of coordinates is the result of a single coarsening, thus typically we have about 2000/c nodes per processor, where c is coarsening ratio (typically, c > 5). So, in most cases, as you've noticed, we start repartitioning process with fewer than 200, maybe even fewer than 100 nodes per processor, which really stresses the partitioner. We have plans to play around with these numbers, but have not done this so far.
After Zoltan2 produced its partitioning, we by default do a bipartite matching to minimize moving of the data. This may conflict with pre-migration procedure that you described as essentially this may move the data twice. I think we found out that moving the data to processors 1-p is suboptimal in many situations, and bipartite graph provides a better matching. However, it may also result in using more nodes than necessary as we don't try to pack the nodes fully. I think we may need to discuss the interaction between two.

mndevec commented 7 years ago

@aprokop The data moves of Zoltan2 only include coordinate data. Both MJ and RCB already migrates the coordinate data during the partitioning, and send them back to the original owners.

For example, if RCB partitions to 16 parts by recursively partitioning to 2. The data is migrated after each recursion. What I did adds one extra level of migration.

MueLu remains oblivious to the data movements of RCB & MJ.

jjellio commented 7 years ago

@mndevec I've taken my Trilinos source on Cori, which is still locked at a version from June. I rolledback your first MJ patch (the one you sent via email), and then cherry-picked the related commit here. Does that sound good? I try to avoid pulling full Trilinos updates from Develop, because they usually result in other things changing - which makes tracking down issues harder.

So, is merging this commit sufficient ??

jhux2 commented 7 years ago

@aprokop wrote:

After Zoltan2 produced its partitioning, we by default do a bipartite matching to minimize moving of the data.

@mndevec wrote:

MueLu remains oblivious to the data movements of RCB & MJ.

@mndevec First off, thanks, this option appears really promising. As I understand it, the premigration is done within Zoltan2 itself, and is really just a way of minimizing the stress of the MJ calculation. The result from Zoltan2 is still a map between DOFs and partition numbers. MueLu is still free to decide where the partitions should be placed, correct? Zoltan2 has its own remap strategy. Would it work with the new premigration strategy?

mndevec commented 7 years ago

@jhux2 Yes, your summary is all correct. Basically MJ will return the exact same result (except some tie-breaking differences) with and without premigration.

mndevec commented 7 years ago

@jjellio Yes, that sounds okay. You can simply replace Zoltan2_AlgMultijagged.hpp file as well.

jjellio commented 7 years ago

Great. I've got it building, the queues on Cori

mndevec commented 7 years ago

@jjellio

Just to make sure, you have both of the commits fd654d2ca6b24a99f04771ae45f2dc540875064e and 6cf5f4aca6adf904234f1eac8769736beac1c9f1 , right?

jjellio commented 7 years ago

No, I did not have fd654d2, only those referenced above.

I've pull it in.

jjellio commented 7 years ago

@mndevec Are there specific output I can look for to ensure the settings above are in effect?

mndevec commented 7 years ago

@jjellio If you set both below, the timers will print Premigration Postmigration times. These times won't be there if you dont set the mj_premigration_option.

       <Parameter name="mj_premigration_option" type="int" value="1"/>
       <Parameter name="timer_type" type="string" value="macro_timers"/>

jjellio commented 7 years ago

Annoyingly, I've had 4 jobs fail on Cori now. Something seems a little off with their filesystem (jobs are hanging because they are failing to load the binary from disk). I am still working on it though... hopefully Cori will actually run something. I have Trilinos + Driver compiled, only waiting for their job system at this point.

pwxy commented 7 years ago

"mj_premigration_option"=1 does change iteration count and convergence history (not surprising because partitioning is slightly different)

"mj_premigration_option"=1 significantly reduces MJ time

2.2B row drekar Poisson problem run on 131072 MPI processes of BG/Q (16 cores per compute node). 27-point stencil, 27 nnz/row.

"premig"="mj_premigration_option" times: "setup"="NOX Total Preconditioner Construction" time (MueLu setup) "MJ"="MultiJagged - Total" time

premig	setup	MJ
0	12.6	3.5
1	9.3	0.143
16	9.3	0.140

"mj_premigration_option"=1 reduced MJ time by 25x "mj_premigration_option"=1 reduced MueLu setup time by 22% for the Drekar Poisson problem

1.8B row drekar MHD problem run on 131072 MPI processes of BG/Q (16 cores per compute node). 27 point stencil, but 8 PDEs/mesh node so ~216 nnz/row. Mesh has ~231M nodes.

MueLu setup and MJ times are per Newton step

premig	setup	MJ
0	29.7	3.8
1	25.5	0.179
16	25.4	0.174

"mj_premigration_option"=1 reduced MJ time by 21x "mj_premigration_option"=1 reduced MueLu setup time by 14%

"mj_premigration_option"=1 significantly reduces MJ time @mndevec thanks for the big reduction in MJ time

aprokop commented 7 years ago

Are there situations when mj_premigration_option=1 is slower than 0? If not, we can set it to 1 by default in MueLu.

mndevec commented 7 years ago

@pwxy I am glad it helped a bit. But I could not understand the second result. Assuming that setup time includes MJ times, MJ got 25 seconds faster, but overall setup got 4 seconds faster? If that is the case, something might be buggy, I will check that a bit more.

@aprokop I think it is safe to have that parameter as default. It might be more expensive if the number of parts is close to the number of processors. For example, lets say #ranks is 1024, and #parts is 1000. Then moving coordinates to 1000 processors will not give much advantage, migration will not pay off. But I suspect this is ever the case in MueLu. But I will try to look at the metrics to detect when such migration won't help, so that I can avoid doing that migration even when premigration is set.

mndevec commented 7 years ago

@pwxy : Partitioning differences happen when tasks have same coordinates, and a partitioning needs to be done along that coordinate. The processors with lower ranks assigns their task on the cut to the part that is left of the cut until the balance is reached. When I premigrate the coordinates using modular arithmetic, that order changes.

I am a bit confused with the timings in the second table. If the new method is increasing the number of iterations, I can change the migration scheme to keep the order of the coordinates as similar to initial case as possible.

pwxy commented 7 years ago

@mndevec sorry, my mistake, i reported the total MJ time for 7 Newton steps in the second table. I have divided the numbers by 7 and corrected the table.

jjellio commented 7 years ago

Seems like the Cori jobs did not complete. This does not seem to be a Trilinos issue. (The jobs failed because srun failed to load the executable from disk). Sigh. I've requeued the job.

srajama1 commented 7 years ago

@jhux2 : Will these improvements translate to Nalu as well ?

pwxy commented 7 years ago

@srajama1 the Z2 improvements definitely would translate to Nalu

jhux2 commented 7 years ago

@srajama1 wrote:

Will these improvements translate to Nalu as well ?

For large enough problems, yes.

srajama1 commented 7 years ago

One stone, >2 birds etc :)

jjellio commented 7 years ago

Cori finally behaved.

I performed 3 runs, and then compared against the data I gathered earlier this year.

These are 512 node runs with flat MPI (serial exec space).

The columns are labeled using short names: A: ./Laplace3D-BS-1-1968x1968x1968_Constructor-Only_MueLu-repartition-mj-1_numsteps-20_Serial_np-32768_decomp-512x64x1x1.yaml B: ./Laplace3D-BS-1-1968x1968x1968_Constructor-Only_MueLu-repartition-mj-16_numsteps-20_Serial_np-32768_decomp-512x64x1x1.yaml C: ./Laplace3D-BS-1-1968x1968x1968_Constructor-Only_MueLu-repartition-mj-64_numsteps-20_Serial_np-32768_decomp-512x64x1x1.yaml

Or: A: mj value = 1 B: mj value = 16 C: mj value = 64

Since these are flat MPI, there are 64 procs per node.

It's worth noting: These runs are faster than my prior runs, as I have isolated some job launch/system parameters that improved overall performance. (mainly network and memory parameters)

First, analysis with respect to the Teuchos Min over Procs

Timer Name	minT	minT_A	minT_speedup_A	minT_B	minT_speedup_B	minT_C	minT_speedup_C
3 - Constructing Preconditioner	133.203	113.866	1.17	107.881	1.235	110.243	1.208
Ifpack2::Chebyshev::compute	12.0688	12.6032	0.958	11.9641	1.009	12.4817	0.967
MueLu: AggregationPhase1Algorithm: BuildAggregates (total)	1.11788	1.12127	0.997	1.12105	0.997	1.12076	0.997
MueLu: AggregationPhase2aAlgorithm: BuildAggregates (total)	0.306487	0.306927	0.999	0.306919	0.999	0.306656	0.999
MueLu: AggregationPhase2bAlgorithm: BuildAggregates (total)	0.585811	0.584812	1.002	0.585032	1.001	0.585052	1.001
MueLu: AggregationPhase3Algorithm: BuildAggregates (total)	0.0191035	0.0191135	0.999	0.0190973	1.0	0.0191076	1.0
MueLu: AmalgamationFactory: Build	0.00478196	0.00476432	1.004	0.00474238	1.008	0.00473309	1.01
MueLu: AmalgamationFactory: Build (level=0)	0.00104189	0.00110126	0.946	0.00107145	0.972	0.00106716	0.976
MueLu: AmalgamationFactory: Build (level=1)	0.00110006	0.00111675	0.985	0.00111818	0.984	0.00109339	1.006
MueLu: AmalgamationFactory: Build (total)	0.00615788	0.00606108	1.016	0.00603676	1.02	0.006042	1.019
MueLu: AmalgamationFactory: Build (total, level=0)	0.00192428	0.00196218	0.981	0.00193954	0.992	0.00192547	0.999
MueLu: AmalgamationFactory: Build (total, level=1)	0.00202847	0.00200391	1.012	0.00200725	1.011	0.00199676	1.016
MueLu: CoalesceDropFactory: Build	1.17216	1.35532	0.865	1.24323	0.943	1.21176	0.967
MueLu: CoalesceDropFactory: Build (level=0)	0.952052	1.076	0.885	0.951133	1.001	0.950995	1.001
MueLu: CoalesceDropFactory: Build (level=1)	0.206779	0.272333	0.759	0.256298	0.807	0.246994	0.837
MueLu: CoalesceDropFactory: Build (total)	1.17921	1.36215	0.866	1.25363	0.941	1.21862	0.968
MueLu: CoalesceDropFactory: Build (total, level=0)	0.954883	1.07889	0.885	0.953628	1.001	0.953485	1.001
MueLu: CoalesceDropFactory: Build (total, level=1)	0.20983	0.274922	0.763	0.258876	0.811	0.249551	0.841
MueLu: CoarseMapFactory: Build	0.0579884	0.0708489	0.818	0.0424592	1.366	0.0435004	1.333
MueLu: CoarseMapFactory: Build (level=0)	0.0236139	0.0473588	0.499	0.0209141	1.129	0.0202892	1.164
MueLu: CoarseMapFactory: Build (level=1)	0.0238459	0.0181882	1.311	0.0170937	1.395	0.0189605	1.258
MueLu: CoarseMapFactory: Build (total)	0.0598254	0.0725987	0.824	0.0442462	1.352	0.045218	1.323
MueLu: CoarseMapFactory: Build (total, level=0)	0.0251658	0.0488493	0.515	0.0224969	1.119	0.0218282	1.153
MueLu: CoarseMapFactory: Build (total, level=1)	0.0250993	0.0193832	1.295	0.0181556	1.382	0.020026	1.253
MueLu: CoordinatesTransferFactory: Build	0.255851	0.321592	0.796	0.269875	0.948	0.26459	0.967
MueLu: CoordinatesTransferFactory: Build (level=1)	0.20692	0.267749	0.773	0.232408	0.89	0.228794	0.904
MueLu: CoordinatesTransferFactory: Build (level=2)	0.0296373	0.0304449	0.973	0.0289249	1.025	0.0285378	1.039
MueLu: CoordinatesTransferFactory: Build (total)	0.257778	0.323497	0.797	0.271753	0.949	0.266546	0.967
MueLu: CoordinatesTransferFactory: Build (total, level=1)	0.208223	0.269093	0.774	0.233754	0.891	0.230195	0.905
MueLu: CoordinatesTransferFactory: Build (total, level=2)	0.0311201	0.0318997	0.976	0.0303884	1.024	0.0300047	1.037
MueLu: FilteredAFactory: Matrix filtering	0.0135632	0.0135062	1.004	0.0134265	1.01	0.0134809	1.006
MueLu: FilteredAFactory: Matrix filtering (level=0)	0.0067668	0.00673556	1.005	0.00666666	1.015	0.00672555	1.006
MueLu: FilteredAFactory: Matrix filtering (level=1)	0.00702453	0.00695515	1.01	0.00698519	1.006	0.00701833	1.001
MueLu: FilteredAFactory: Matrix filtering (total)	1.19326	1.37684	0.867	1.27578	0.935	1.2336	0.967
MueLu: FilteredAFactory: Matrix filtering (total, level=0)	0.962846	1.08793	0.885	0.960433	1.003	0.960279	1.003
MueLu: FilteredAFactory: Matrix filtering (total, level=1)	0.218047	0.28224	0.773	0.266366	0.819	0.257054	0.848
MueLu: Hierarchy: Setup (total)	106.665	94.6524	1.127	91.0813	1.171	92.5787	1.152
MueLu: Hierarchy: Setup (total, level=0)	9.23364	9.86529	0.936	9.19923	1.004	9.69397	0.953
MueLu: Hierarchy: Setup (total, level=1)	65.2539	54.0457	1.207	52.6619	1.239	52.8497	1.235
MueLu: Hierarchy: Setup (total, level=2)	32.1703	30.7156	1.047	29.2035	1.102	30.0185	1.072
MueLu: Ifpack2Smoother: Setup Smoother	12.0837	12.6179	0.958	11.9795	1.009	12.4969	0.967
MueLu: Ifpack2Smoother: Setup Smoother (level=0)	9.17177	9.79658	0.936	9.13721	1.004	9.63259	0.952
MueLu: Ifpack2Smoother: Setup Smoother (level=1)	2.90732	2.81604	1.032	2.83689	1.025	2.85884	1.017
MueLu: Ifpack2Smoother: Setup Smoother (total)	12.0852	12.6194	0.958	11.9811	1.009	12.4985	0.967
MueLu: Ifpack2Smoother: Setup Smoother (total, level=0)	9.17283	9.79761	0.936	9.13825	1.004	9.63367	0.952
MueLu: Ifpack2Smoother: Setup Smoother (total, level=1)	2.90826	2.81696	1.032	2.83781	1.025	2.85977	1.017
MueLu: NullspaceFactory: Nullspace factory	0.0022049	0.00212812	1.036	0.0021534	1.024	0.00213838	1.031
MueLu: NullspaceFactory: Nullspace factory (level=1)	0.000804901	0.000773668	1.04	0.000772476	1.042	0.000775337	1.038
MueLu: NullspaceFactory: Nullspace factory (total)	0.00295568	0.00284481	1.039	0.00286913	1.03	0.00286674	1.031
MueLu: NullspaceFactory: Nullspace factory (total, level=1)	0.0017457	0.00166631	1.048	0.00168109	1.038	0.001683	1.037
MueLu: PreserveDirichletAggregationAlgorithm: BuildAggregates (total)	0.0138421	0.0138633	0.998	0.013855	0.999	0.0138443	1.0
MueLu: RAPFactory: Computing Ac	50.0265	39.0137	1.282	38.8377	1.288	39.6967	1.26
MueLu: RAPFactory: Computing Ac (level=1)	41.7596	30.503	1.369	30.2115	1.382	30.8922	1.352
MueLu: RAPFactory: Computing Ac (level=2)	8.26646	8.51049	0.971	8.62572	0.958	8.80365	0.939
MueLu: RAPFactory: Computing Ac (total)	74.4773	63.825	1.167	62.4261	1.193	62.8734	1.185
MueLu: RAPFactory: Computing Ac (total, level=1)	61.7013	50.6002	1.219	49.2123	1.254	49.3627	1.25
MueLu: RAPFactory: Computing Ac (total, level=2)	12.7716	13.2128	0.967	13.1635	0.97	13.4966	0.946
MueLu: RAPFactory: MxM: A x P (sub, total)	30.8866	19.5175	1.583	19.3713	1.594	19.6371	1.573
MueLu: RAPFactory: MxM: A x P (sub, total, level=1)	26.3952	15.0185	1.758	14.6961	1.796	14.8945	1.772
MueLu: RAPFactory: MxM: A x P (sub, total, level=2)	4.34689	4.37113	0.994	4.52926	0.96	4.59841	0.945
MueLu: RAPFactory: MxM: P' x (AP) (implicit) (sub, total)	17.9064	18.2992	0.979	18.2348	0.982	18.7395	0.956
MueLu: RAPFactory: MxM: P' x (AP) (implicit) (sub, total, level=1)	14.7008	14.8643	0.989	14.9069	0.986	15.263	0.963
MueLu: RAPFactory: MxM: P' x (AP) (implicit) (sub, total, level=2)	3.12038	3.35583	0.93	3.30213	0.945	3.44295	0.906
MueLu: RAPFactory: Projections (sub, total)	0.268961	0.334426	0.804	0.283305	0.949	0.278298	0.966
MueLu: RAPFactory: Projections (sub, total, level=1)	0.214407	0.275365	0.779	0.2402	0.893	0.236699	0.906
MueLu: RAPFactory: Projections (sub, total, level=2)	0.0373669	0.0382051	0.978	0.0367229	1.018	0.0362492	1.031
MueLu: RebalanceAcFactory: Computing Ac	1.47886	1.28836	1.148	1.37442	1.076	1.47519	1.002
MueLu: RebalanceAcFactory: Computing Ac (level=1)	0.000919104	0.000914097	1.005	0.000906706	1.014	0.000906229	1.014
MueLu: RebalanceAcFactory: Computing Ac (level=2)	1.47509	1.28461	1.148	1.37073	1.076	1.47149	1.002
MueLu: RebalanceAcFactory: Computing Ac (total)	1.48033	1.28974	1.148	1.37581	1.076	1.47664	1.002
MueLu: RebalanceAcFactory: Computing Ac (total, level=1)	0.00159764	0.00157857	1.012	0.00157309	1.016	0.00156879	1.018
MueLu: RebalanceAcFactory: Computing Ac (total, level=2)	1.47609	1.28557	1.148	1.37181	1.076	1.47254	1.002
MueLu: RebalanceAcFactory: Rebalancing existing Ac (sub, total)	1.473	1.28244	1.149	1.3687	1.076	1.46943	1.002
MueLu: RebalanceAcFactory: Rebalancing existing Ac (sub, total, level=2)	1.47231	1.28181	1.149	1.36809	1.076	1.46884	1.002
MueLu: RebalanceTransferFactory: Build	1.86853	0.662105	2.822	0.587661	3.18	0.584206	3.198
MueLu: RebalanceTransferFactory: Build (level=1)	0.00861192	0.0085187	1.011	0.00858736	1.003	0.00861168	1.0
MueLu: RebalanceTransferFactory: Build (level=2)	1.85869	0.652593	2.848	0.578334	3.214	0.574637	3.235
MueLu: RebalanceTransferFactory: Build (total)	92.8745	80.5075	1.154	77.5063	1.198	78.3871	1.185
MueLu: RebalanceTransferFactory: Build (total, level=1)	62.2292	51.1232	1.217	49.7156	1.252	49.882	1.248
MueLu: RebalanceTransferFactory: Build (total, level=2)	30.639	29.3794	1.043	27.7797	1.103	28.4936	1.075
MueLu: RebalanceTransferFactory: Rebalancing coordinates (sub, total)	1.36365	0.29074	4.69	0.279353	4.881	0.281746	4.84
MueLu: RebalanceTransferFactory: Rebalancing coordinates (sub, total, level=2)	1.36301	0.290126	4.698	0.278701	4.891	0.281103	4.849
MueLu: RebalanceTransferFactory: Rebalancing nullspace (sub, total)	0.478152	0.345351	1.385	0.28307	1.689	0.276604	1.729
MueLu: RebalanceTransferFactory: Rebalancing nullspace (sub, total, level=2)	0.468191	0.307752	1.521	0.279171	1.677	0.275468	1.7
MueLu: RebalanceTransferFactory: Rebalancing prolongator (sub, total)	0.00154638	0.0016346	0.946	0.00164032	0.943	0.00158954	0.973
MueLu: RebalanceTransferFactory: Rebalancing prolongator (sub, total, level=1)	0.000156641	0.000157833	0.992	0.000153065	1.023	0.000155449	1.008
MueLu: RebalanceTransferFactory: Rebalancing prolongator (sub, total, level=2)	0.000175714	0.000161409	1.089	0.000153303	1.146	0.000159502	1.102
MueLu: RepartitionFactory: Build	4.01112	2.90198	1.382	2.77668	1.445	2.78348	1.441
MueLu: RepartitionFactory: Build (level=1)	0.0710549	0.0790834	0.898	0.0648406	1.096	0.0650868	1.092
MueLu: RepartitionFactory: Build (level=2)	3.94348	2.82917	1.394	2.71806	1.451	2.7267	1.446
MueLu: RepartitionFactory: Build (total)	90.9992	79.8413	1.14	76.9149	1.183	77.7986	1.17
MueLu: RepartitionFactory: Build (total, level=1)	62.2196	51.114	1.217	49.7064	1.252	49.8727	1.248
MueLu: RepartitionFactory: Build (total, level=2)	28.7729	28.7232	1.002	27.1559	1.06	27.9133	1.031
MueLu: RepartitionFactory: DeterminePartitionPlacement (sub, total)	2.16593	1.67578	1.292	1.61149	1.344	1.62484	1.333
MueLu: RepartitionFactory: DeterminePartitionPlacement (sub, total, level=2)	2.1516	1.66523	1.292	1.6107	1.336	1.61715	1.33
MueLu: RepartitionFactory: Import construction (sub, total)	0.178778	0.183483	0.974	0.174675	1.023	0.172039	1.039
MueLu: RepartitionFactory: Import construction (sub, total, level=2)	0.177805	0.182521	0.974	0.173728	1.023	0.171141	1.039
MueLu: RepartitionHeuristicFactory: Build	0.181134	0.162647	1.114	0.10191	1.777	0.10067	1.799
MueLu: RepartitionHeuristicFactory: Build (level=1)	0.0012362	0.00121522	1.017	0.00119829	1.032	0.00119901	1.031
MueLu: RepartitionHeuristicFactory: Build (level=2)	0.175703	0.158279	1.11	0.0975144	1.802	0.0960703	1.829
MueLu: RepartitionHeuristicFactory: Build (total)	0.182834	0.164314	1.113	0.103575	1.765	0.102381	1.786
MueLu: RepartitionHeuristicFactory: Build (total, level=1)	0.0021131	0.00205827	1.027	0.00203872	1.036	0.00204062	1.036
MueLu: RepartitionHeuristicFactory: Build (total, level=2)	0.177002	0.159318	1.111	0.0986276	1.795	0.0971265	1.822
MueLu: SaPFactory: Eigenvalue estimate (sub, total)	0.00232673	0.00236464	0.984	0.00239062	0.973	0.0023911	0.973
MueLu: SaPFactory: Eigenvalue estimate (sub, total, level=1)	0.00048089	0.000488281	0.985	0.000499964	0.962	0.000493288	0.975
MueLu: SaPFactory: Eigenvalue estimate (sub, total, level=2)	0.000529766	0.000547171	0.968	0.00055027	0.963	0.000539303	0.982
MueLu: SaPFactory: Fused (I-omegaD^{-1} A)Ptent (sub, total)	17.4518	17.281	1.01	16.7842	1.04	16.3727	1.066
MueLu: SaPFactory: Fused (I-omegaD^{-1} A)Ptent (sub, total, level=1)	14.7972	14.6195	1.012	14.0903	1.05	13.5937	1.089
MueLu: SaPFactory: Fused (I-omegaD^{-1} A)Ptent (sub, total, level=2)	2.62217	2.62631	0.998	2.66042	0.986	2.7456	0.955
MueLu: SaPFactory: Prolongator smoothing	18.1524	18.0551	1.005	17.4068	1.043	17.0445	1.065
MueLu: SaPFactory: Prolongator smoothing (level=1)	15.1824	14.988	1.013	14.4158	1.053	13.9215	1.091
MueLu: SaPFactory: Prolongator smoothing (level=2)	2.97229	3.06939	0.968	2.99335	0.993	3.12557	0.951
MueLu: SaPFactory: Prolongator smoothing (total)	24.4494	24.8042	0.986	23.5875	1.037	23.1744	1.055
MueLu: SaPFactory: Prolongator smoothing (total, level=1)	19.937	20.0958	0.992	18.9996	1.049	18.4683	1.08
MueLu: SaPFactory: Prolongator smoothing (total, level=2)	4.50363	4.70195	0.958	4.53704	0.993	4.69203	0.96
MueLu: TentativePFactory: Build	2.06574	2.21146	0.934	1.97963	1.043	1.9958	1.035
MueLu: TentativePFactory: Build (level=1)	1.76385	1.89218	0.932	1.68795	1.045	1.67369	1.054
MueLu: TentativePFactory: Build (level=2)	0.304613	0.32258	0.944	0.29579	1.03	0.326294	0.934
MueLu: TentativePFactory: Build (total)	5.10001	5.36983	0.95	4.89904	1.041	4.89529	1.042
MueLu: TentativePFactory: Build (total, level=1)	3.78759	4.01725	0.943	3.62135	1.046	3.5849	1.057
MueLu: TentativePFactory: Build (total, level=2)	1.30965	1.34938	0.971	1.27477	1.027	1.30747	1.002
MueLu: UncoupledAggregationFactory: Algo "Phase - (Dirichlet)" (sub, total)	0.0400887	0.0392182	1.022	0.0382597	1.048	0.0381672	1.05
MueLu: UncoupledAggregationFactory: Algo "Phase - (Dirichlet)" (sub, total, level=0)	0.0248234	0.0252602	0.983	0.024178	1.027	0.0238178	1.042
MueLu: UncoupledAggregationFactory: Algo "Phase - (Dirichlet)" (sub, total, level=1)	0.0127878	0.0115187	1.11	0.0119317	1.072	0.0119495	1.07
MueLu: UncoupledAggregationFactory: Algo "Phase 1 (main)" (sub, total)	1.24981	1.34895	0.927	1.23654	1.011	1.24532	1.004
MueLu: UncoupledAggregationFactory: Algo "Phase 1 (main)" (sub, total, level=0)	0.892242	0.961197	0.928	0.877354	1.017	0.885417	1.008
MueLu: UncoupledAggregationFactory: Algo "Phase 1 (main)" (sub, total, level=1)	0.355059	0.385654	0.921	0.357448	0.993	0.358019	0.992
MueLu: UncoupledAggregationFactory: Algo "Phase 2a (secondary)" (sub, total)	0.432495	0.444781	0.972	0.432148	1.001	0.430427	1.005
MueLu: UncoupledAggregationFactory: Algo "Phase 2a (secondary)" (sub, total, level=0)	0.237578	0.244671	0.971	0.23999	0.99	0.239665	0.991
MueLu: UncoupledAggregationFactory: Algo "Phase 2a (secondary)" (sub, total, level=1)	0.192826	0.198099	0.973	0.189759	1.016	0.189069	1.02
MueLu: UncoupledAggregationFactory: Algo "Phase 2b (expansion)" (sub, total)	1.07983	1.09312	0.988	1.01176	1.067	0.978378	1.104
MueLu: UncoupledAggregationFactory: Algo "Phase 2b (expansion)" (sub, total, level=0)	0.724284	0.73154	0.99	0.658897	1.099	0.629304	1.151
MueLu: UncoupledAggregationFactory: Algo "Phase 2b (expansion)" (sub, total, level=1)	0.353173	0.35938	0.983	0.350775	1.007	0.347042	1.018
MueLu: UncoupledAggregationFactory: Algo "Phase 3 (cleanup)" (sub, total)	0.0513871	0.0453174	1.134	0.0449564	1.143	0.0440938	1.165
MueLu: UncoupledAggregationFactory: Algo "Phase 3 (cleanup)" (sub, total, level=0)	0.0348258	0.0319204	1.091	0.0304883	1.142	0.0296364	1.175
MueLu: UncoupledAggregationFactory: Algo "Phase 3 (cleanup)" (sub, total, level=1)	0.0142105	0.0112979	1.258	0.012181	1.167	0.0119438	1.19
MueLu: UncoupledAggregationFactory: Build	2.96683	3.0775	0.964	2.86712	1.035	2.84723	1.042
MueLu: UncoupledAggregationFactory: Build (level=0)	1.9913	2.07202	0.961	1.90649	1.044	1.88686	1.055
MueLu: UncoupledAggregationFactory: Build (level=1)	0.971516	1.00236	0.969	0.957369	1.015	0.957104	1.015
MueLu: UncoupledAggregationFactory: Build (total)	2.9684	3.07905	0.964	2.86864	1.035	2.8487	1.042
MueLu: UncoupledAggregationFactory: Build (total, level=0)	1.99238	2.07308	0.961	1.90753	1.044	1.88791	1.055
MueLu: UncoupledAggregationFactory: Build (total, level=1)	0.972592	1.00338	0.969	0.95842	1.015	0.958149	1.015
MueLu: Zoltan2Interface: Build	11.8289	12.4761	0.948	11.1307	1.063	11.5472	1.024
MueLu: Zoltan2Interface: Build (level=1)	0.00154781	0.00157189	0.985	0.00153112	1.011	0.00155973	0.992
MueLu: Zoltan2Interface: Build (level=2)	11.8236	12.4709	0.948	11.1258	1.063	11.5423	1.024
MueLu: Zoltan2Interface: Build (total)	11.8305	12.4776	0.948	11.1322	1.063	11.5487	1.024
MueLu: Zoltan2Interface: Build (total, level=1)	0.00236654	0.00238228	0.993	0.00233459	1.014	0.00237465	0.997
MueLu: Zoltan2Interface: Build (total, level=2)	11.825	12.4722	0.948	11.1271	1.063	11.5436	1.024
MueLu: Zoltan2Interface: Zoltan2 multijagged (sub, total)	11.5944	12.3101	0.942	10.9699	1.057	11.3904	1.018
MueLu: Zoltan2Interface: Zoltan2 multijagged (sub, total, level=2)	11.5928	12.3091	0.942	10.969	1.057	11.3892	1.018
MultiJagged - Final DistributorPlanComm		0.0		0.0		0.0
MultiJagged - Final DistributorPlanCreating		0.0		0.0		0.0
MultiJagged - Migration DistributorMigration-0		0.0		0.0		0.0
MultiJagged - Migration DistributorMigration-1		0.0		0.0		0.0
MultiJagged - Migration DistributorPlanCreating-0		0.0		0.0		0.0
MultiJagged - Migration DistributorPlanCreating-1		0.0		0.0		0.0
MultiJagged - Part_Assignment		0.0		0.0		0.0
MultiJagged - PostMigration DistributorMigration		0.00521994		0.00521016		0.00522637
MultiJagged - PostMigration DistributorPlanCreating		0.070869		0.0473049		0.0442643
MultiJagged - PreMigration DistributorMigration		0.00152302		0.00151134		0.00150895
MultiJagged - PreMigration DistributorPlanCreating		0.138623		0.102562		0.149993
MultiJagged - Problem_Free		0.0		0.0		0.0
MultiJagged - Problem_Migration-0		0.0		0.0		0.0
MultiJagged - Problem_Migration-1		0.0		0.0		0.0
MultiJagged - Problem_Partitioning		0.0		0.0		0.0
MultiJagged - Problem_Partitioning_0		0.0		0.0		0.0
MultiJagged - Problem_Partitioning_1		0.0		0.0		0.0
MultiJagged - Problem_Partitioning_2		0.0		0.0		0.0
MultiJagged - Solution_Part_Assignment		0.0		0.0		0.0
MultiJagged - Total		0.0		0.0		0.0
Tpetra MueLu::A*P-1: ESFC-G-Maps	9.41753e-05	8.89301e-05	1.059	8.67844e-05	1.085	9.08375e-05	1.037
Tpetra MueLu::A*P-1: ESFC-G-Setup	0.00011301	0.000102043	1.107	0.000104427	1.082	0.000103712	1.09
Tpetra MueLu::A*P-1: ESFC-G-cGC (noconst)	0.72014	0.721942	0.998	0.722394	0.997	0.722159	0.997
Tpetra MueLu::A*P-1: ESFC-G-cIS	4.43459e-05	3.79086e-05	1.17	3.71933e-05	1.192	3.8147e-05	1.163
Tpetra MueLu::A*P-1: ESFC-G-fLG	0.000468731	0.000465393	1.007	0.000460148	1.019	0.000448704	1.045
Tpetra MueLu::A*P-1: ESFC-G-mIXcheckE	6.67572e-05	6.55651e-05	1.018	6.50883e-05	1.026	6.48499e-05	1.029
Tpetra MueLu::A*P-1: ESFC-G-mIXcheckI	0.000132561	0.000125408	1.057	0.000124216	1.067	0.000126123	1.051
Tpetra MueLu::A*P-1: ESFC-G-mIXmake	0.814445	0.686175	1.187	0.228652	3.562	0.276377	2.947
Tpetra MueLu::A*P-1: ESFC-M-Graph	1.58008	1.4764	1.07	1.04423	1.513	1.09702	1.44
Tpetra MueLu::A*P-1: ESFC-M-cGC	0.000130892	0.000126839	1.032	0.000131845	0.993	0.000129938	1.007
Tpetra MueLu::A*P-1: ESFC-M-cIS	3.17097e-05	3.45707e-05	0.917	3.60012e-05	0.881	3.33786e-05	0.95
Tpetra MueLu::A*P-1: ESFC-M-fLGAM	0.000285625	0.000305414	0.935	0.000317812	0.899	0.000294447	0.97
Tpetra MueLu::A*P-1: TAFC CreateImporter	0.712661	0.793083	0.899	0.881244	0.809	1.02291	0.697
Tpetra MueLu::A*P-1: TAFC ESFC	0.0393136	0.0390835	1.006	0.0393136	1.0	0.0391588	1.004
Tpetra MueLu::A*P-1: TAFC ImportSetup	0.00185871	0.0018127	1.025	0.00183439	1.013	0.00184298	1.009
Tpetra MueLu::A*P-1: TAFC Pack-1	0.0384784	0.0395095	0.974	0.0340366	1.131	0.0364583	1.055
Tpetra MueLu::A*P-1: TAFC Pack-2	0.232871	0.233187	0.999	0.233024	0.999	0.233482	0.997
Tpetra MueLu::A*P-1: TAFC Transfer	0.142509	0.168558	0.845	0.142966	0.997	0.140026	1.018
Tpetra MueLu::A*P-1: TAFC Unpack-1	0.101846	0.101355	1.005	0.101338	1.005	0.101167	1.007
Tpetra MueLu::A*P-1: TAFC Unpack-2	0.52976	0.516627	1.025	0.522953	1.013	0.543844	0.974
Tpetra MueLu::A*P-1: TAFC Unpack-3	0.0274577	0.0274096	1.002	0.0275388	0.997	0.0274272	1.001
Tpetra MueLu::A*P-2: ESFC-G-Maps	9.36985e-05	8.79765e-05	1.065	8.65459e-05	1.083	8.82149e-05	1.062
Tpetra MueLu::A*P-2: ESFC-G-Setup	9.799e-05	9.15527e-05	1.07	8.82149e-05	1.111	9.10759e-05	1.076
Tpetra MueLu::A*P-2: ESFC-G-cGC (noconst)	0.0800033	0.0801256	0.998	0.0799646	1.0	0.0801008	0.999
Tpetra MueLu::A*P-2: ESFC-G-cIS	4.17233e-05	3.40939e-05	1.224	3.74317e-05	1.115	3.60012e-05	1.159
Tpetra MueLu::A*P-2: ESFC-G-fLG	0.000462532	0.000442505	1.045	0.000437498	1.057	0.000436783	1.059
Tpetra MueLu::A*P-2: ESFC-G-mIXcheckE	6.48499e-05	6.29425e-05	1.03	6.17504e-05	1.05	6.24657e-05	1.038
Tpetra MueLu::A*P-2: ESFC-G-mIXcheckI	0.00012064	0.000115395	1.045	0.000114679	1.052	0.000113249	1.065
Tpetra MueLu::A*P-2: ESFC-G-mIXmake	0.000166416	0.000163794	1.016	0.000159264	1.045	0.000159264	1.045
Tpetra MueLu::A*P-2: ESFC-M-Graph	0.0814302	0.0815868	0.998	0.0812991	1.002	0.0815198	0.999
Tpetra MueLu::A*P-2: ESFC-M-cGC	0.0001266	0.000124216	1.019	0.000127554	0.993	0.000126839	0.998
Tpetra MueLu::A*P-2: ESFC-M-cIS	3.24249e-05	3.43323e-05	0.944	3.33786e-05	0.971	3.43323e-05	0.944
Tpetra MueLu::A*P-2: ESFC-M-fLGAM	0.000281811	0.000295162	0.955	0.000294447	0.957	0.00029707	0.949
Tpetra MueLu::A*P-2: TAFC CreateImporter	0.190487	0.268868	0.708	0.241769	0.788	0.25767	0.739
Tpetra MueLu::A*P-2: TAFC ESFC	0.0195765	0.0197988	0.989	0.0195923	0.999	0.0199852	0.98
Tpetra MueLu::A*P-2: TAFC ImportSetup	0.000403404	0.000389338	1.036	0.000389814	1.035	0.000398397	1.013
Tpetra MueLu::A*P-2: TAFC Pack-1	0.00826216	0.00822306	1.005	0.00822544	1.004	0.00823975	1.003
Tpetra MueLu::A*P-2: TAFC Pack-2	0.0956204	0.0955856	1.0	0.0957384	0.999	0.0957873	0.998
Tpetra MueLu::A*P-2: TAFC Transfer	0.298163	0.334769	0.891	0.29632	1.006	0.25454	1.171
Tpetra MueLu::A*P-2: TAFC Unpack-1	0.0581758	0.0574939	1.012	0.0575116	1.012	0.0576386	1.009
Tpetra MueLu::A*P-2: TAFC Unpack-2	0.336292	0.357312	0.941	0.366248	0.918	0.370468	0.908
Tpetra MueLu::A*P-2: TAFC Unpack-3	0.025949	0.0259457	1.0	0.0257988	1.006	0.0256526	1.012
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-Maps	8.96454e-05	8.55923e-05	1.047	8.60691e-05	1.042	8.51154e-05	1.053
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-Setup	0.000147104	0.000136137	1.081	0.000139952	1.051	0.000141621	1.039
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-cGC (const)	0.102267	0.105866	0.966	0.0997784	1.025	0.0996923	1.026
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-cIS	4.45843e-05	4.14848e-05	1.075	4.17233e-05	1.069	4.17233e-05	1.069
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-fLG	0.000653028	0.000641346	1.018	0.000640392	1.02	0.000642776	1.016
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-mIXcheckE	6.77109e-05	6.48499e-05	1.044	6.46114e-05	1.048	6.55651e-05	1.033
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-mIXcheckI	0.000132084	0.000115633	1.142	0.000119448	1.106	0.000123501	1.069
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-mIXmake	0.606233	1.02781	0.59	0.677964	0.894	0.779699	0.778
Tpetra MueLu::R*(AP)-implicit-1: ESFC-M-Graph	0.716526	1.14856	0.624	0.785431	0.912	0.886295	0.808
Tpetra MueLu::R*(AP)-implicit-1: ESFC-M-cGC	0.000137091	0.000128269	1.069	0.000128508	1.067	0.000128269	1.069
Tpetra MueLu::R*(AP)-implicit-1: ESFC-M-cIS	3.26633e-05	3.57628e-05	0.913	3.52859e-05	0.926	3.60012e-05	0.907
Tpetra MueLu::R*(AP)-implicit-1: ESFC-M-fLGAM	0.000335932	0.000351667	0.955	0.000344992	0.974	0.000333548	1.007
Tpetra MueLu::R*(AP)-implicit-1: TAFC CreateImporter	1.08499	1.06718	1.017	1.30846	0.829	1.45952	0.743
Tpetra MueLu::R*(AP)-implicit-1: TAFC ESFC	0.0802662	0.0867822	0.925	0.076879	1.044	0.0768275	1.045
Tpetra MueLu::R*(AP)-implicit-1: TAFC ImportSetup	0.00183797	0.0017941	1.024	0.00180697	1.017	0.00181079	1.015
Tpetra MueLu::R*(AP)-implicit-1: TAFC Pack-1	0.000922918	0.000910997	1.013	0.00090003	1.025	0.000903606	1.021
Tpetra MueLu::R*(AP)-implicit-1: TAFC Pack-2	0.0638883	0.0635598	1.005	0.0635028	1.006	0.0632765	1.01
Tpetra MueLu::R*(AP)-implicit-1: TAFC Transfer	0.0708659	0.119436	0.593	0.0736964	0.962	0.0668187	1.061
Tpetra MueLu::R*(AP)-implicit-1: TAFC Unpack-1	1.04253	1.04935	0.994	1.05098	0.992	1.04278	1.0
Tpetra MueLu::R*(AP)-implicit-1: TAFC Unpack-2	1.05979	1.13122	0.937	1.16291	0.911	1.12898	0.939
Tpetra MueLu::R*(AP)-implicit-1: TAFC Unpack-3	0.787594	0.815028	0.966	0.78727	1.0	0.798074	0.987
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-Maps	0.000136614	0.000135899	1.005	0.000134468	1.016	0.000131845	1.036
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-Setup	0.000154018	0.000156879	0.982	0.000149488	1.03	0.000143766	1.071
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-cGC (noconst)	0.100589	0.101322	0.993	0.100988	0.996	0.101198	0.994
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-cIS	4.07696e-05	3.6478e-05	1.118	3.55244e-05	1.148	3.48091e-05	1.171
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-fLG	0.00041008	0.000388145	1.057	0.000396013	1.036	0.000401974	1.02
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-mIXcheckE	9.9659e-05	0.00010848	0.919	0.000106812	0.933	0.00010252	0.972
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-mIXcheckI	8.24928e-05	7.89165e-05	1.045	7.7486e-05	1.065	7.67708e-05	1.075
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-mIXmake	0.0680482	0.131947	0.516	0.0539329	1.262	0.0549889	1.237
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-M-Graph	0.198511	0.251539	0.789	0.170783	1.162	0.175395	1.132
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-M-cGC	0.000124454	0.000117779	1.057	0.000118732	1.048	0.000118732	1.048
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-M-cIS	3.00407e-05	3.17097e-05	0.947	3.05176e-05	0.984	3.21865e-05	0.933
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-M-fLGAM	0.000260115	0.000277042	0.939	0.000275612	0.944	0.000278711	0.933
Tpetra MueLu::R*(AP)-implicit-1XP: : Transpose Local	0.859184	0.998963	0.86	0.93805	0.916	0.937888	0.916
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-Maps	9.01222e-05	8.7738e-05	1.027	8.65459e-05	1.041	8.46386e-05	1.065
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-Setup	9.72748e-05	9.10759e-05	1.068	8.74996e-05	1.112	8.67844e-05	1.121
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-cGC (const)	0.0245152	0.0184844	1.326	0.0198874	1.233	0.0198164	1.237
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-cIS	4.62532e-05	4.31538e-05	1.072	4.33922e-05	1.066	4.24385e-05	1.09
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-fLG	0.000653028	0.000693798	0.941	0.000647068	1.009	0.000684023	0.955
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-mIXcheckE	6.81877e-05	6.41346e-05	1.063	6.4373e-05	1.059	6.36578e-05	1.071
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-mIXcheckI	0.00012064	0.000114441	1.054	0.000111818	1.079	0.000112295	1.074
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-mIXmake	0.356901	0.548435	0.651	0.367948	0.97	0.424449	0.841
Tpetra MueLu::R*(AP)-implicit-2: ESFC-M-Graph	0.389921	0.574345	0.679	0.402456	0.969	0.448525	0.869
Tpetra MueLu::R*(AP)-implicit-2: ESFC-M-cGC	0.000138044	0.000129223	1.068	0.000128031	1.078	0.0001297	1.064
Tpetra MueLu::R*(AP)-implicit-2: ESFC-M-cIS	3.6478e-05	4.02927e-05	0.905	3.93391e-05	0.927	3.98159e-05	0.916
Tpetra MueLu::R*(AP)-implicit-2: ESFC-M-fLGAM	0.000383854	0.000408411	0.94	0.00041008	0.936	0.000409603	0.937
Tpetra MueLu::R*(AP)-implicit-2: TAFC CreateImporter	0.27224	0.378851	0.719	0.34781	0.783	0.326031	0.835
Tpetra MueLu::R*(AP)-implicit-2: TAFC ESFC	0.0179904	0.0167196	1.076	0.015553	1.157	0.0149355	1.205
Tpetra MueLu::R*(AP)-implicit-2: TAFC ImportSetup	0.000367641	0.000356674	1.031	0.000365496	1.006	0.000357389	1.029
Tpetra MueLu::R*(AP)-implicit-2: TAFC Pack-1	0.000990868	0.000970125	1.021	0.000975132	1.016	0.000977755	1.013
Tpetra MueLu::R*(AP)-implicit-2: TAFC Pack-2	0.012799	0.012574	1.018	0.0126038	1.015	0.0125797	1.017
Tpetra MueLu::R*(AP)-implicit-2: TAFC Transfer	0.128101	0.145106	0.883	0.125493	1.021	0.127919	1.001
Tpetra MueLu::R*(AP)-implicit-2: TAFC Unpack-1	0.0619235	0.0616293	1.005	0.0615118	1.007	0.0617719	1.002
Tpetra MueLu::R*(AP)-implicit-2: TAFC Unpack-2	0.274702	0.294784	0.932	0.313516	0.876	0.33374	0.823
Tpetra MueLu::R*(AP)-implicit-2: TAFC Unpack-3	0.0758474	0.0757558	1.001	0.0758772	1.0	0.0758924	0.999
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-Maps	0.000132561	0.000125885	1.053	0.000126362	1.049	0.00012517	1.059
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-Setup	0.000143528	0.00013566	1.058	0.000136852	1.049	0.000132561	1.083
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-cGC (noconst)	0.00393057	0.00387359	1.015	0.00382686	1.027	0.00390172	1.007
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-cIS	4.91142e-05	4.1008e-05	1.198	4.00543e-05	1.226	4.17233e-05	1.177
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-fLG	0.000371218	0.000365734	1.015	0.000329018	1.128	0.000353813	1.049
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-mIXcheckE	9.9659e-05	0.000102997	0.968	0.000105143	0.948	0.000102282	0.974
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-mIXcheckI	7.55787e-05	7.24792e-05	1.043	7.39098e-05	1.023	7.29561e-05	1.036
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-mIXmake	0.000185251	0.000181675	1.02	0.000181437	1.021	0.000183105	1.012
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-M-Graph	0.00539327	0.00520062	1.037	0.00523424	1.03	0.00529099	1.019
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-M-cGC	0.000124931	0.000119448	1.046	0.000118732	1.052	0.000118494	1.054
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-M-cIS	3.74317e-05	4.17233e-05	0.897	4.05312e-05	0.924	3.88622e-05	0.963
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-M-fLGAM	0.000293016	0.000323534	0.906	0.000309706	0.946	0.000305891	0.958
Tpetra MueLu::R*(AP)-implicit-2XP: : Transpose Local	0.0948594	0.0948861	1.0	0.0946581	1.002	0.0947428	1.001
Tpetra MueLu::RebalanceAc-2: TAFC ImportSetup	0.0229416	0.0227211	1.01	0.0215464	1.065	0.0225105	1.019
Tpetra MueLu::RebalanceAc-2: TAFC Pack-1	0.410979	0.306371	1.341	0.279625	1.47	0.277499	1.481
Tpetra MueLu::RebalanceAc-2: TAFC Pack-2	0.000674009	0.000674009	1.0	0.00069356	0.972	0.0006814	0.989
Tpetra MueLu::RebalanceAc-2: TAFC Transfer	0.0187562	0.0208874	0.898	0.0187674	0.999	0.0171342	1.095
Tpetra MueLu::RebalanceAc-2: TAFC Unpack-1	0.000946999	0.000913143	1.037	0.000917196	1.032	0.000910759	1.04
Tpetra MueLu::RebalanceAc-2: TAFC Unpack-2	0.25005	0.376424	0.664	0.393214	0.636	0.394051	0.635
Tpetra MueLu::SaP-1: ESFC-G-Maps	9.39369e-05	9.17912e-05	1.023	9.08375e-05	1.034	9.01222e-05	1.042
Tpetra MueLu::SaP-1: ESFC-G-Setup	0.000113726	0.000103474	1.099	0.000105619	1.077	0.000104904	1.084
Tpetra MueLu::SaP-1: ESFC-G-cGC (noconst)	0.660388	0.660698	1.0	0.660746	0.999	0.660493	1.0
Tpetra MueLu::SaP-1: ESFC-G-cIS	3.98159e-05	3.33786e-05	1.193	3.40939e-05	1.168	3.31402e-05	1.201
Tpetra MueLu::SaP-1: ESFC-G-fLG	0.000438213	0.000422001	1.038	0.000424862	1.031	0.000420809	1.041
Tpetra MueLu::SaP-1: ESFC-G-mIXcheckE	6.74725e-05	6.58035e-05	1.025	6.46114e-05	1.044	6.55651e-05	1.029
Tpetra MueLu::SaP-1: ESFC-G-mIXcheckI	0.000130892	0.000128031	1.022	0.000121832	1.074	0.000121355	1.079
Tpetra MueLu::SaP-1: ESFC-G-mIXmake	0.217969	0.396664	0.55	0.0725286	3.005	0.0599625	3.635
Tpetra MueLu::SaP-1: ESFC-M-Graph	0.921509	1.10406	0.835	0.787805	1.17	0.771978	1.194
Tpetra MueLu::SaP-1: ESFC-M-cGC	0.000130415	0.000120401	1.083	0.000119925	1.087	0.00012064	1.081
Tpetra MueLu::SaP-1: ESFC-M-cIS	3.02792e-05	3.24249e-05	0.934	3.21865e-05	0.941	3.29018e-05	0.92
Tpetra MueLu::SaP-1: ESFC-M-fLGAM	0.000269413	0.000286102	0.942	0.000282764	0.953	0.000281334	0.958
Tpetra MueLu::SaP-1: TAFC CreateImporter	0.2527	0.283468	0.891	0.359554	0.703	0.282502	0.895
Tpetra MueLu::SaP-1: TAFC ESFC	0.0355051	0.0357378	0.993	0.0358675	0.99	0.0356238	0.997
Tpetra MueLu::SaP-1: TAFC ImportSetup	0.000909567	0.000902414	1.008	0.000880957	1.032	0.000895739	1.015
Tpetra MueLu::SaP-1: TAFC Pack-1	0.0355823	0.0374	0.951	0.0507803	0.701	0.0342283	1.04
Tpetra MueLu::SaP-1: TAFC Pack-2	0.193182	0.192189	1.005	0.192087	1.006	0.192583	1.003
Tpetra MueLu::SaP-1: TAFC Transfer	0.104791	0.111477	0.94	0.113295	0.925	0.101844	1.029
Tpetra MueLu::SaP-1: TAFC Unpack-1	0.0775661	0.0761738	1.018	0.0761309	1.019	0.0766158	1.012
Tpetra MueLu::SaP-1: TAFC Unpack-2	0.204813	0.246936	0.829	0.231853	0.883	0.198116	1.034
Tpetra MueLu::SaP-1: TAFC Unpack-3	0.00569057	0.00562119	1.012	0.00562525	1.012	0.00559926	1.016
Tpetra MueLu::SaP-2: ESFC-G-Maps	9.36985e-05	8.51154e-05	1.101	8.79765e-05	1.065	8.74996e-05	1.071
Tpetra MueLu::SaP-2: ESFC-G-Setup	9.70364e-05	9.2268e-05	1.052	8.86917e-05	1.094	9.05991e-05	1.071
Tpetra MueLu::SaP-2: ESFC-G-cGC (noconst)	0.0718484	0.0720994	0.997	0.0721483	0.996	0.0720842	0.997
Tpetra MueLu::SaP-2: ESFC-G-cIS	4.1008e-05	3.6478e-05	1.124	3.8147e-05	1.075	3.57628e-05	1.147
Tpetra MueLu::SaP-2: ESFC-G-fLG	0.0004282	0.000415564	1.03	0.000417709	1.025	0.000414133	1.034
Tpetra MueLu::SaP-2: ESFC-G-mIXcheckE	6.4373e-05	6.27041e-05	1.027	6.22272e-05	1.034	6.22272e-05	1.034
Tpetra MueLu::SaP-2: ESFC-G-mIXcheckI	0.000121117	0.000113726	1.065	0.000111818	1.083	0.000109911	1.102
Tpetra MueLu::SaP-2: ESFC-G-mIXmake	0.000162363	0.000164032	0.99	0.000163317	0.994	0.000158548	1.024
Tpetra MueLu::SaP-2: ESFC-M-Graph	0.0732863	0.0734599	0.998	0.0734327	0.998	0.0734317	0.998
Tpetra MueLu::SaP-2: ESFC-M-cGC	0.000130415	0.000118732	1.098	0.000120163	1.085	0.000118256	1.103
Tpetra MueLu::SaP-2: ESFC-M-cIS	3.43323e-05	3.6478e-05	0.941	3.6478e-05	0.941	3.62396e-05	0.947
Tpetra MueLu::SaP-2: ESFC-M-fLGAM	0.000291109	0.000302315	0.963	0.000303984	0.958	0.000300884	0.968
Tpetra MueLu::SaP-2: TAFC CreateImporter	0.212773	0.226392	0.94	0.159321	1.335	0.216405	0.983
Tpetra MueLu::SaP-2: TAFC ESFC	0.0185511	0.0184715	1.004	0.0185652	0.999	0.0184896	1.003
Tpetra MueLu::SaP-2: TAFC ImportSetup	0.000302315	0.000287771	1.051	0.000289917	1.043	0.000293016	1.032
Tpetra MueLu::SaP-2: TAFC Pack-1	0.00797081	0.00793433	1.005	0.00796533	1.001	0.00794625	1.003
Tpetra MueLu::SaP-2: TAFC Pack-2	0.0692873	0.0693965	0.998	0.0692995	1.0	0.0693262	0.999
Tpetra MueLu::SaP-2: TAFC Transfer	0.151609	0.15362	0.987	0.151312	1.002	0.153179	0.99
Tpetra MueLu::SaP-2: TAFC Unpack-1	0.0362072	0.0357053	1.014	0.0357952	1.012	0.0362096	1.0
Tpetra MueLu::SaP-2: TAFC Unpack-2	0.23024	0.227623	1.011	0.252518	0.912	0.275864	0.835
Tpetra MueLu::SaP-2: TAFC Unpack-3	0.00310946	0.0030973	1.004	0.00307846	1.01	0.00310493	1.001
Tpetra MueLu::TentativeP-1: ESFC-G-Maps	0.000132322	0.000138521	0.955	0.000124216	1.065	0.000123262	1.074
Tpetra MueLu::TentativeP-1: ESFC-G-Setup	0.000154734	0.000154734	1.0	0.000138283	1.119	0.000136375	1.135
Tpetra MueLu::TentativeP-1: ESFC-G-cGC (noconst)	0.386555	0.386822	0.999	0.386717	1.0	0.386633	1.0
Tpetra MueLu::TentativeP-1: ESFC-G-cIS	5.4121e-05	4.76837e-05	1.135	4.62532e-05	1.17	4.93526e-05	1.097
Tpetra MueLu::TentativeP-1: ESFC-G-fLG	0.000456572	0.000436544	1.046	0.000442028	1.033	0.000455618	1.002
Tpetra MueLu::TentativeP-1: ESFC-G-mIXcheckE	7.00951e-05	7.51019e-05	0.933	7.39098e-05	0.948	7.27177e-05	0.964
Tpetra MueLu::TentativeP-1: ESFC-G-mIXcheckI	8.46386e-05	7.82013e-05	1.082	7.67708e-05	1.102	7.77245e-05	1.089
Tpetra MueLu::TentativeP-1: ESFC-G-mIXmake	0.133555	0.295763	0.452	0.17135	0.779	0.179388	0.745
Tpetra MueLu::TentativeP-1: ESFC-M-Graph	0.543122	0.699296	0.777	0.573611	0.947	0.58104	0.935
Tpetra MueLu::TentativeP-1: ESFC-M-cGC	0.000128031	0.00012064	1.061	0.000126123	1.015	0.000125885	1.017
Tpetra MueLu::TentativeP-1: ESFC-M-cIS	3.38554e-05	3.60012e-05	0.94	3.62396e-05	0.934	3.6478e-05	0.928
Tpetra MueLu::TentativeP-1: ESFC-M-fLGAM	0.000266314	0.000272989	0.976	0.000285625	0.932	0.000271559	0.981
Tpetra MueLu::TentativeP-2: ESFC-G-Maps	0.000125647	0.000117064	1.073	0.000117064	1.073	0.000117064	1.073
Tpetra MueLu::TentativeP-2: ESFC-G-Setup	0.000125408	0.000118256	1.06	0.000113726	1.103	0.000114441	1.096
Tpetra MueLu::TentativeP-2: ESFC-G-cGC (noconst)	0.0470788	0.047184	0.998	0.0471399	0.999	0.047183	0.998
Tpetra MueLu::TentativeP-2: ESFC-G-cIS	4.24385e-05	3.93391e-05	1.079	3.95775e-05	1.072	3.98159e-05	1.066
Tpetra MueLu::TentativeP-2: ESFC-G-fLG	0.000348091	0.000358105	0.972	0.000336409	1.035	0.000357151	0.975
Tpetra MueLu::TentativeP-2: ESFC-G-mIXcheckE	6.81877e-05	6.96182e-05	0.979	6.8903e-05	0.99	6.96182e-05	0.979
Tpetra MueLu::TentativeP-2: ESFC-G-mIXcheckI	7.62939e-05	7.4625e-05	1.022	7.53403e-05	1.013	7.58171e-05	1.006
Tpetra MueLu::TentativeP-2: ESFC-G-mIXmake	0.000178576	0.000176668	1.011	0.000172853	1.033	0.000172377	1.036
Tpetra MueLu::TentativeP-2: ESFC-M-Graph	0.0483625	0.0485458	0.996	0.0483358	1.001	0.0484269	0.999
Tpetra MueLu::TentativeP-2: ESFC-M-cGC	0.000115395	0.000109434	1.054	0.000110865	1.041	0.000110626	1.043
Tpetra MueLu::TentativeP-2: ESFC-M-cIS	3.43323e-05	3.83854e-05	0.894	3.79086e-05	0.906	3.74317e-05	0.917
Tpetra MueLu::TentativeP-2: ESFC-M-fLGAM	0.000289917	0.000329494	0.88	0.000324488	0.893	0.000326157	0.889
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-Maps	9.5129e-05	9.08375e-05	1.047	9.20296e-05	1.034	9.01222e-05	1.056
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-Setup	0.000139475	0.000131607	1.06	0.000130653	1.068	0.000133753	1.043
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-cGC (noconst)	0.0353985	0.0352244	1.005	0.035428	0.999	0.0353258	1.002
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-cIS	4.64916e-05	3.88622e-05	1.196	3.74317e-05	1.242	3.95775e-05	1.175
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-fLG	0.000453234	0.0004282	1.058	0.000438929	1.033	0.000425339	1.066
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-mIXcheckE	6.29425e-05	6.17504e-05	1.019	6.17504e-05	1.019	6.27041e-05	1.004
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-mIXcheckI	9.799e-05	9.89437e-05	0.99	9.46522e-05	1.035	9.84669e-05	0.995
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-mIXmake	0.000147343	0.000143051	1.03	0.000142574	1.033	0.000143766	1.025
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-M-Graph	0.0370595	0.036849	1.006	0.0370269	1.001	0.0369661	1.003
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-M-cGC	0.000137091	0.000129461	1.059	0.0001266	1.083	0.000128269	1.069
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-M-cIS	3.40939e-05	3.55244e-05	0.96	3.31402e-05	1.029	3.38554e-05	1.007
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-M-fLGAM	0.000286341	0.000302792	0.946	0.000290155	0.987	0.000291348	0.983
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-Maps	9.94205e-05	0.000100136	0.993	9.77516e-05	1.017	0.000100613	0.988
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-Setup	0.000116348	0.000102043	1.14	0.000106096	1.097	0.000107288	1.084
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-cGC (noconst)	0.0161059	0.0163012	0.988	0.0161846	0.995	0.0163963	0.982
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-cIS	5.10216e-05	4.48227e-05	1.138	4.55379e-05	1.12	4.41074e-05	1.157
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-fLG	0.000413656	0.000391006	1.058	0.000392914	1.053	0.000394583	1.048
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-mIXcheckE	6.48499e-05	6.22272e-05	1.042	6.19888e-05	1.046	6.24657e-05	1.038
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-mIXcheckI	0.000100613	0.000100851	0.998	0.000101089	0.995	0.000101089	0.995
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-mIXmake	0.000153303	0.00014782	1.037	0.000146151	1.049	0.000148296	1.034
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-M-Graph	0.0175462	0.0177486	0.989	0.0175629	0.999	0.0178277	0.984
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-M-cGC	0.000139236	0.000126123	1.104	0.000127077	1.096	0.000126123	1.104
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-M-cIS	3.88622e-05	4.29153e-05	0.906	4.29153e-05	0.906	4.17233e-05	0.931
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-M-fLGAM	0.000308037	0.000321865	0.957	0.000316858	0.972	0.000324726	0.949
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-Maps	9.29832e-05	9.13143e-05	1.018	9.13143e-05	1.018	8.89301e-05	1.046
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-Setup	0.000112772	0.000109434	1.031	0.000109911	1.026	0.000110388	1.022
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-cGC (const)	0.076139	0.0826616	0.921	0.0729136	1.044	0.0727844	1.046
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-cIS	4.36306e-05	4.22001e-05	1.034	4.19617e-05	1.04	4.12464e-05	1.058
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-fLG	0.000705719	0.000679016	1.039	0.000673056	1.049	0.000669956	1.053
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-mIXcheckE	6.19888e-05	6.1512e-05	1.008	6.12736e-05	1.012	6.10352e-05	1.016
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-mIXcheckI	9.91821e-05	0.000101089	0.981	9.98974e-05	0.993	0.000101089	0.981
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-mIXmake	0.000152349	0.000143528	1.061	0.00014472	1.053	0.00014329	1.063
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-M-Graph	0.0779738	0.0844767	0.923	0.0746529	1.044	0.0744972	1.047
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-M-cGC	0.000137329	0.000127554	1.077	0.000128031	1.073	0.000130653	1.051
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-M-cIS	3.26633e-05	3.60012e-05	0.907	3.6478e-05	0.895	3.60012e-05	0.907
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-M-fLGAM	0.000329256	0.000373602	0.881	0.000350952	0.938	0.000354767	0.928
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-Maps	9.41753e-05	8.72612e-05	1.079	9.17912e-05	1.026	9.01222e-05	1.045
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-Setup	0.00011301	0.000106335	1.063	0.000106096	1.065	0.000105619	1.07
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-cGC (const)	0.014122	0.0127113	1.111	0.011524	1.225	0.0110509	1.278
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-cIS	4.26769e-05	3.98159e-05	1.072	4.02927e-05	1.059	3.83854e-05	1.112
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-fLG	0.000529528	0.000476599	1.111	0.000499249	1.061	0.000488281	1.084
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-mIXcheckE	6.4373e-05	6.34193e-05	1.015	6.24657e-05	1.031	6.29425e-05	1.023
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-mIXcheckI	9.77516e-05	0.000100851	0.969	0.000100851	0.969	0.000100136	0.976
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-mIXmake	0.000156641	0.000147104	1.065	0.000148296	1.056	0.000144005	1.088
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-M-Graph	0.015734	0.0143573	1.096	0.0132432	1.188	0.0126686	1.242
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-M-cGC	0.0001266	0.000113964	1.111	0.000115633	1.095	0.000115395	1.097
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-M-cIS	3.62396e-05	3.98159e-05	0.91	4.00543e-05	0.905	3.93391e-05	0.921
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-M-fLGAM	0.00033617	0.000396729	0.847	0.000391483	0.859	0.000398636	0.843
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-Maps	0.000104904	0.000104189	1.007	0.000102997	1.019	0.000100851	1.04
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-Setup	0.000121593	0.000116587	1.043	0.000117302	1.037	0.00011611	1.047
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-cGC (noconst)	0.0319095	0.0322206	0.99	0.0323715	0.986	0.0321989	0.991
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-cIS	4.81606e-05	4.05312e-05	1.188	3.83854e-05	1.255	4.00543e-05	1.202
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-fLG	0.000417709	0.000382662	1.092	0.000399828	1.045	0.000406504	1.028
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-mIXcheckE	6.60419e-05	6.55651e-05	1.007	6.65188e-05	0.993	6.58035e-05	1.004
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-mIXcheckI	0.00010705	0.000111818	0.957	0.00010848	0.987	0.000110149	0.972
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-mIXmake	0.000169992	0.000164032	1.036	0.000162601	1.045	0.000161886	1.05
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-M-Graph	0.0334437	0.033622	0.995	0.0337949	0.99	0.0335886	0.996
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-M-cGC	0.000146151	0.000130653	1.119	0.000130653	1.119	0.000130653	1.119
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-M-cIS	3.52859e-05	3.76701e-05	0.937	3.6478e-05	0.967	3.69549e-05	0.955
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-M-fLGAM	0.000286102	0.000312328	0.916	0.000307083	0.932	0.0002985	0.958
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-Maps	9.94205e-05	9.67979e-05	1.027	9.67979e-05	1.027	9.70364e-05	1.025
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-Setup	0.000118017	0.000108004	1.093	0.000105858	1.115	0.000107765	1.095
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-cGC (noconst)	0.0152795	0.0151539	1.008	0.0152242	1.004	0.0151346	1.01
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-cIS	4.14848e-05	3.69549e-05	1.123	3.88622e-05	1.067	3.71933e-05	1.115
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-fLG	0.000297546	0.000288725	1.031	0.000297546	1.0	0.000293732	1.013
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-mIXcheckE	6.48499e-05	6.22272e-05	1.042	6.34193e-05	1.023	6.19888e-05	1.046
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-mIXcheckI	9.9659e-05	0.000101089	0.986	0.000101805	0.979	0.000100613	0.991
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-mIXmake	0.000157833	0.000150681	1.047	0.000151634	1.041	0.000151396	1.043
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-M-Graph	0.0165956	0.0164306	1.01	0.0165076	1.005	0.0164609	1.008
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-M-cGC	0.000114202	0.000108242	1.055	0.000107288	1.064	0.000110865	1.03
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-M-cIS	3.57628e-05	3.88622e-05	0.92	3.79086e-05	0.943	3.83854e-05	0.932
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-M-fLGAM	0.000272751	0.000287771	0.948	0.000286102	0.953	0.00028491	0.957
TpetraExt MueLu::A*P-1: MMM All I&X	3.11609	3.15193	0.989	3.34563	0.931	3.55526	0.876
TpetraExt MueLu::A*P-1: MMM All Multiply	23.2689	11.8431	1.965	11.3422	2.052	11.3241	2.055
TpetraExt MueLu::A*P-1: MMM All Setup	0.000287056	0.000296831	0.967	0.000296593	0.968	0.000293255	0.979
TpetraExt MueLu::A*P-1: MMM I&X Alloc	0.000161886	0.000156403	1.035	0.000151873	1.066	0.000154257	1.049
TpetraExt MueLu::A*P-1: MMM I&X Collective-0	0.00380087	0.00409317	0.929	0.00442934	0.858	0.00352407	1.079
TpetraExt MueLu::A*P-1: MMM I&X Import-2	0.0293009	0.0292172	1.003	0.0293877	0.997	0.0289278	1.013
TpetraExt MueLu::A*P-1: MMM I&X Import-3	2.72212	2.80466	0.971	3.02008	0.901	3.23973	0.84
TpetraExt MueLu::A*P-1: MMM I&X Import-4	0.000626802	0.000597239	1.049	0.000593901	1.055	0.000618219	1.014
TpetraExt MueLu::A*P-1: MMM I&X RemoteMap	0.326274	0.27446	1.189	0.267296	1.221	0.266534	1.224
TpetraExt MueLu::A*P-1: MMM M5 Cmap	23.2672	11.8412	1.965	11.3403	2.052	11.3221	2.055
TpetraExt MueLu::A*P-1: MMM Newmatrix ESFC	1.58254	1.51463	1.045	1.04704	1.511	1.09968	1.439
TpetraExt MueLu::A*P-1: MMM Newmatrix Final Sort	1.02391	1.05582	0.97	1.02942	0.995	1.02968	0.994
TpetraExt MueLu::A*P-1: MMM Newmatrix SerialCore	4.33128	4.44593	0.974	4.31453	1.004	4.27496	1.013
TpetraExt MueLu::A*P-2: MMM All I&X	1.72634	1.72599	1.0	1.87654	0.92	1.98581	0.869
TpetraExt MueLu::A*P-2: MMM All Multiply	2.61638	2.64057	0.991	2.64751	0.988	2.60766	1.003
TpetraExt MueLu::A*P-2: MMM All Setup	0.000267744	0.000251293	1.065	0.000271082	0.988	0.000256777	1.043
TpetraExt MueLu::A*P-2: MMM I&X Alloc	0.000168085	0.000165939	1.013	0.000163317	1.029	0.000164509	1.022
TpetraExt MueLu::A*P-2: MMM I&X Collective-0	0.00339556	0.003824	0.888	0.00428128	0.793	0.00365973	0.928
TpetraExt MueLu::A*P-2: MMM I&X Import-2	0.0123026	0.0123372	0.997	0.0122452	1.005	0.012387	0.993
TpetraExt MueLu::A*P-2: MMM I&X Import-3	1.61595	1.64916	0.98	1.79694	0.899	1.90501	0.848
TpetraExt MueLu::A*P-2: MMM I&X Import-4	0.000597715	0.000583172	1.025	0.000585318	1.021	0.000581026	1.029
TpetraExt MueLu::A*P-2: MMM I&X RemoteMap	0.0634198	0.0472534	1.342	0.0468488	1.354	0.0480173	1.321
TpetraExt MueLu::A*P-2: MMM M5 Cmap	2.61428	2.63855	0.991	2.64537	0.988	2.60555	1.003
TpetraExt MueLu::A*P-2: MMM Newmatrix ESFC	0.0836587	0.0838208	0.998	0.0834174	1.003	0.0838614	0.998
TpetraExt MueLu::A*P-2: MMM Newmatrix Final Sort	0.293043	0.294827	0.994	0.293712	0.998	0.293528	0.998
TpetraExt MueLu::A*P-2: MMM Newmatrix SerialCore	1.97044	1.97721	0.997	1.97714	0.997	1.97627	0.997
TpetraExt MueLu::R*(AP)-implicit-1: MMM All I&X	9.48906e-05	9.58443e-05	0.99	9.32217e-05	1.018	9.46522e-05	1.003
TpetraExt MueLu::R*(AP)-implicit-1: MMM All Multiply	14.6967	14.8598	0.989	14.9027	0.986	15.2589	0.963
TpetraExt MueLu::R*(AP)-implicit-1: MMM All Setup	0.000230312	0.000226021	1.019	0.0002141	1.076	0.000216961	1.062
TpetraExt MueLu::R*(AP)-implicit-1: MMM I&X Alloc	9.82285e-05	8.67844e-05	1.132	8.91685e-05	1.102	8.84533e-05	1.111
TpetraExt MueLu::R*(AP)-implicit-1: MMM M5 Cmap	7.9462	8.31077	0.956	8.05067	0.987	8.16836	0.973
TpetraExt MueLu::R*(AP)-implicit-1: MMM Newmatrix ESFC	0.718917	1.15091	0.625	0.787817	0.913	0.888674	0.809
TpetraExt MueLu::R*(AP)-implicit-1: MMM Newmatrix Final Sort	0.643103	0.648316	0.992	0.644184	0.998	0.654164	0.983
TpetraExt MueLu::R*(AP)-implicit-1: MMM Newmatrix SerialCore	3.38996	3.38676	1.001	3.37065	1.006	3.36974	1.006
TpetraExt MueLu::R*(AP)-implicit-1: MMM-T AB-core	7.94717	8.31179	0.956	8.05161	0.987	8.16932	0.973
TpetraExt MueLu::R*(AP)-implicit-1: MMM-T I&X	0.000713825	0.000692844	1.03	0.000690222	1.034	0.000686884	1.039
TpetraExt MueLu::R*(AP)-implicit-1: MMM-T Transpose	0.860161	1.00003	0.86	0.939014	0.916	0.938856	0.916
TpetraExt MueLu::R*(AP)-implicit-1: MMM-T exportAndFillComplete	5.87967	5.52694	1.064	5.89494	0.997	6.13469	0.958
TpetraExt MueLu::R*(AP)-implicit-2: MMM All I&X	8.7738e-05	8.63075e-05	1.017	8.82149e-05	0.995	8.79765e-05	0.997
TpetraExt MueLu::R*(AP)-implicit-2: MMM All Multiply	3.11741	3.35309	0.93	3.29913	0.945	3.43993	0.906
TpetraExt MueLu::R*(AP)-implicit-2: MMM All Setup	0.000191212	0.000186443	1.026	0.000184298	1.038	0.000188589	1.014
TpetraExt MueLu::R*(AP)-implicit-2: MMM I&X Alloc	0.000110149	0.000106573	1.034	0.000108242	1.018	0.000103951	1.06
TpetraExt MueLu::R*(AP)-implicit-2: MMM M5 Cmap	1.65694	1.83316	0.904	1.66259	0.997	1.70904	0.97
TpetraExt MueLu::R*(AP)-implicit-2: MMM Newmatrix ESFC	0.392336	0.576648	0.68	0.404681	0.969	0.450838	0.87
TpetraExt MueLu::R*(AP)-implicit-2: MMM Newmatrix Final Sort	0.0765419	0.0766265	0.999	0.0767159	0.998	0.0766585	0.998
TpetraExt MueLu::R*(AP)-implicit-2: MMM Newmatrix SerialCore	0.868533	0.871345	0.997	0.870037	0.998	0.872258	0.996
TpetraExt MueLu::R*(AP)-implicit-2: MMM-T AB-core	1.65818	1.83447	0.904	1.66387	0.997	1.71036	0.969
TpetraExt MueLu::R*(AP)-implicit-2: MMM-T I&X	0.00083971	0.000857115	0.98	0.000842333	0.997	0.000856161	0.981
TpetraExt MueLu::R*(AP)-implicit-2: MMM-T Transpose	0.0955746	0.0958304	0.997	0.0954323	1.001	0.0955293	1.0
TpetraExt MueLu::R*(AP)-implicit-2: MMM-T exportAndFillComplete	1.33929	1.39914	0.957	1.51523	0.884	1.60969	0.832
TpetraExt MueLu::SaP-1: Jacobi All I&X	2.59245	2.84642	0.911	3.06104	0.847	2.61553	0.991
TpetraExt MueLu::SaP-1: Jacobi All Multiply	7.43263	6.54744	1.135	6.18132	1.202	6.15875	1.207
TpetraExt MueLu::SaP-1: Jacobi All Setup	0.000314713	0.000315428	0.998	0.000300646	1.047	0.000303507	1.037
TpetraExt MueLu::SaP-1: Jacobi M5 Cmap	0.0751557	0.07774	0.967	0.0893631	0.841	0.076112	0.987
TpetraExt MueLu::SaP-1: Jacobi Newmatrix ESFC	0.924056	1.11677	0.827	0.790064	1.17	0.774468	1.193
TpetraExt MueLu::SaP-1: Jacobi Newmatrix Final Sort	0.383594	0.393793	0.974	0.386534	0.992	0.386292	0.993
TpetraExt MueLu::SaP-1: Jacobi Newmatrix SerialCore	4.05055	4.09344	0.99	4.01491	1.009	4.00231	1.012
TpetraExt MueLu::SaP-1: MMM I&X Alloc	0.000363827	0.000353336	1.03	0.000347853	1.046	0.000346184	1.051
TpetraExt MueLu::SaP-1: MMM I&X Collective-0	0.00831437	0.00893211	0.931	0.0149953	0.554	0.00682902	1.218
TpetraExt MueLu::SaP-1: MMM I&X Import-2	0.0288167	0.0294652	0.978	0.0293396	0.982	0.028806	1.0
TpetraExt MueLu::SaP-1: MMM I&X Import-3	1.34511	1.41613	0.95	1.81284	0.742	1.37897	0.975
TpetraExt MueLu::SaP-1: MMM I&X Import-4	0.000559568	0.00053668	1.043	0.000567913	0.985	0.000571728	0.979
TpetraExt MueLu::SaP-1: MMM I&X RemoteMap	1.1879	1.37328	0.865	1.17148	1.014	1.16649	1.018
TpetraExt MueLu::SaP-2: Jacobi All I&X	1.21994	1.17654	1.037	1.24129	0.983	1.32483	0.921
TpetraExt MueLu::SaP-2: Jacobi All Multiply	0.983584	0.983772	1.0	0.983418	1.0	0.983587	1.0
TpetraExt MueLu::SaP-2: Jacobi All Setup	0.00022459	0.000227928	0.985	0.000228882	0.981	0.000221014	1.016
TpetraExt MueLu::SaP-2: Jacobi M5 Cmap	0.0195336	0.0178354	1.095	0.0176897	1.104	0.0170529	1.145
TpetraExt MueLu::SaP-2: Jacobi Newmatrix ESFC	0.0755336	0.0756512	0.998	0.0756357	0.999	0.0756438	0.999
TpetraExt MueLu::SaP-2: Jacobi Newmatrix Final Sort	0.0868857	0.087095	0.998	0.0869365	0.999	0.0869484	0.999
TpetraExt MueLu::SaP-2: Jacobi Newmatrix SerialCore	0.777189	0.7788	0.998	0.778639	0.998	0.778721	0.998
TpetraExt MueLu::SaP-2: MMM I&X Alloc	0.000328302	0.000317812	1.033	0.000317812	1.033	0.00031662	1.037
TpetraExt MueLu::SaP-2: MMM I&X Collective-0	0.00816274	0.00813508	1.003	0.00686264	1.189	0.00753355	1.084
TpetraExt MueLu::SaP-2: MMM I&X Import-2	0.0120356	0.0122375	0.984	0.0121748	0.989	0.012183	0.988
TpetraExt MueLu::SaP-2: MMM I&X Import-3	1.10996	1.05811	1.049	1.14088	0.973	1.22358	0.907
TpetraExt MueLu::SaP-2: MMM I&X Import-4	0.00052166	0.000510454	1.022	0.000522852	0.998	0.000511885	1.019
TpetraExt MueLu::SaP-2: MMM I&X RemoteMap	0.0704103	0.0880499	0.8	0.0597854	1.178	0.0587277	1.199
create problem		0.0612752		0.0560253		0.0522726
create solution		0.0198317		0.0178964		0.0177751

Max over Procs

Timer Name	maxT	maxT_A	maxT_speedup_A	maxT_B	maxT_speedup_B	maxT_C	maxT_speedup_C
3 - Constructing Preconditioner	133.397	113.885	1.171	107.898	1.236	110.273	1.21
Ifpack2::Chebyshev::compute	13.6357	14.1861	0.961	13.5028	1.01	14.0207	0.973
MueLu: AggregationPhase1Algorithm: BuildAggregates (total)	1.28241	1.30126	0.986	1.26318	1.015	1.2628	1.016
MueLu: AggregationPhase2aAlgorithm: BuildAggregates (total)	0.45832	0.455343	1.007	0.454601	1.008	0.454629	1.008
MueLu: AggregationPhase2bAlgorithm: BuildAggregates (total)	0.848099	0.833629	1.017	0.833266	1.018	0.832878	1.018
MueLu: AggregationPhase3Algorithm: BuildAggregates (total)	0.0222416	0.0224895	0.989	0.0223839	0.994	0.0223091	0.997
MueLu: AmalgamationFactory: Build	0.0133731	0.0127325	1.05	0.0129857	1.03	0.0130696	1.023
MueLu: AmalgamationFactory: Build (level=0)	0.00163031	0.0016408	0.994	0.00153685	1.061	0.00149512	1.09
MueLu: AmalgamationFactory: Build (level=1)	0.00177836	0.00175214	1.015	0.00150943	1.178	0.00155354	1.145
MueLu: AmalgamationFactory: Build (total)	0.0168951	0.0160949	1.05	0.016361	1.033	0.0163777	1.032
MueLu: AmalgamationFactory: Build (total, level=0)	0.00257397	0.00259495	0.992	0.00242567	1.061	0.00239897	1.073
MueLu: AmalgamationFactory: Build (total, level=1)	0.00281453	0.00274754	1.024	0.00247192	1.139	0.00252557	1.114
MueLu: CoalesceDropFactory: Build	1.26771	1.4553	0.871	1.34816	0.94	1.30208	0.974
MueLu: CoalesceDropFactory: Build (level=0)	0.966562	1.08748	0.889	0.957048	1.01	0.961312	1.005
MueLu: CoalesceDropFactory: Build (level=1)	0.216098	0.283674	0.762	0.314873	0.686	0.262983	0.822
MueLu: CoalesceDropFactory: Build (total)	1.28557	1.47217	0.873	1.36535	0.942	1.31937	0.974
MueLu: CoalesceDropFactory: Build (total, level=0)	0.968983	1.08985	0.889	0.959403	1.01	0.963644	1.006
MueLu: CoalesceDropFactory: Build (total, level=1)	0.218593	0.28611	0.764	0.317227	0.689	0.265433	0.824
MueLu: CoarseMapFactory: Build	0.102184	0.243372	0.42	0.0876713	1.166	0.0861142	1.187
MueLu: CoarseMapFactory: Build (level=0)	0.0313911	0.0513854	0.611	0.0258932	1.212	0.0231102	1.358
MueLu: CoarseMapFactory: Build (level=1)	0.0306158	0.0215859	1.418	0.0188396	1.625	0.0208449	1.469
MueLu: CoarseMapFactory: Build (total)	0.106409	0.247349	0.43	0.0917113	1.16	0.0901258	1.181
MueLu: CoarseMapFactory: Build (total, level=0)	0.0326681	0.0527112	0.62	0.0271647	1.203	0.0243809	1.34
MueLu: CoarseMapFactory: Build (total, level=1)	0.0317206	0.0225806	1.405	0.0198674	1.597	0.021878	1.45
MueLu: CoordinatesTransferFactory: Build	0.52203	0.506207	1.031	0.501795	1.04	0.518708	1.006
MueLu: CoordinatesTransferFactory: Build (level=1)	0.452174	0.435548	1.038	0.433746	1.042	0.450425	1.004
MueLu: CoordinatesTransferFactory: Build (level=2)	0.0367854	0.0373189	0.986	0.0356693	1.031	0.0353422	1.041
MueLu: CoordinatesTransferFactory: Build (total)	0.526714	0.510917	1.031	0.506358	1.04	0.523326	1.006
MueLu: CoordinatesTransferFactory: Build (total, level=1)	0.453488	0.436937	1.038	0.435054	1.042	0.451743	1.004
MueLu: CoordinatesTransferFactory: Build (total, level=2)	0.0379834	0.0384719	0.987	0.0368521	1.031	0.0364971	1.041
MueLu: FilteredAFactory: Matrix filtering	0.0382638	0.0381882	1.002	0.0377874	1.013	0.0378468	1.011
MueLu: FilteredAFactory: Matrix filtering (level=0)	0.00799012	0.00909615	0.878	0.0079751	1.002	0.00788498	1.013
MueLu: FilteredAFactory: Matrix filtering (level=1)	0.00834703	0.00816584	1.022	0.00811362	1.029	0.0080955	1.031
MueLu: FilteredAFactory: Matrix filtering (total)	1.32307	1.50958	0.876	1.40273	0.943	1.35664	0.975
MueLu: FilteredAFactory: Matrix filtering (total, level=0)	0.976032	1.09747	0.889	0.966744	1.01	0.970983	1.005
MueLu: FilteredAFactory: Matrix filtering (total, level=1)	0.226115	0.293451	0.771	0.324767	0.696	0.272867	0.829
MueLu: Hierarchy: Setup (total)	131.818	110.844	1.189	106.714	1.235	108.855	1.211
MueLu: Hierarchy: Setup (total, level=0)	9.40457	9.88484	0.951	9.20559	1.022	9.70452	0.969
MueLu: Hierarchy: Setup (total, level=1)	65.443	54.0556	1.211	52.719	1.241	52.8688	1.238
MueLu: Hierarchy: Setup (total, level=2)	33.5328	32.1661	1.042	30.5242	1.099	31.25	1.073
MueLu: Ifpack2Smoother: Setup Smoother	13.6759	14.2265	0.961	13.5428	1.01	14.0616	0.973
MueLu: Ifpack2Smoother: Setup Smoother (level=0)	9.3409	9.80195	0.953	9.14164	1.022	9.6403	0.969
MueLu: Ifpack2Smoother: Setup Smoother (level=1)	2.98284	2.88615	1.034	2.94782	1.012	2.93032	1.018
MueLu: Ifpack2Smoother: Setup Smoother (total)	13.6803	14.2308	0.961	13.5471	1.01	14.0659	0.973
MueLu: Ifpack2Smoother: Setup Smoother (total, level=0)	9.34195	9.80297	0.953	9.14263	1.022	9.64135	0.969
MueLu: Ifpack2Smoother: Setup Smoother (total, level=1)	2.98379	2.88709	1.033	2.94875	1.012	2.9313	1.018
MueLu: NullspaceFactory: Nullspace factory	0.00988531	0.00979972	1.009	0.00960922	1.029	0.00959826	1.03
MueLu: NullspaceFactory: Nullspace factory (level=1)	0.00108075	0.00215864	0.501	0.00100994	1.07	0.00106645	1.013
MueLu: NullspaceFactory: Nullspace factory (total)	0.0132642	0.0129752	1.022	0.0128508	1.032	0.0129099	1.027
MueLu: NullspaceFactory: Nullspace factory (total, level=1)	0.00209188	0.00312185	0.67	0.00199294	1.05	0.00207758	1.007
MueLu: PreserveDirichletAggregationAlgorithm: BuildAggregates (total)	0.0161788	0.0165274	0.979	0.0162184	0.998	0.0166204	0.973
MueLu: RAPFactory: Computing Ac	54.4697	43.2808	1.259	43.2426	1.26	44.157	1.234
MueLu: RAPFactory: Computing Ac (level=1)	41.9879	30.6553	1.37	30.4008	1.381	31.1002	1.35
MueLu: RAPFactory: Computing Ac (level=2)	8.2681	8.51162	0.971	8.62683	0.958	8.8049	0.939
MueLu: RAPFactory: Computing Ac (total)	83.087	70.6263	1.176	69.1849	1.201	69.7365	1.191
MueLu: RAPFactory: Computing Ac (total, level=1)	61.9343	50.7574	1.22	49.4029	1.254	49.5756	1.249
MueLu: RAPFactory: Computing Ac (total, level=2)	12.7792	13.2262	0.966	13.222	0.967	13.5136	0.946
MueLu: RAPFactory: MxM: A x P (sub, total)	34.2485	22.4458	1.526	22.3836	1.53	22.701	1.509
MueLu: RAPFactory: MxM: A x P (sub, total, level=1)	26.6394	15.2257	1.75	14.9505	1.782	15.1812	1.755
MueLu: RAPFactory: MxM: A x P (sub, total, level=2)	4.8749	4.89486	0.996	5.05402	0.965	5.12372	0.951
MueLu: RAPFactory: MxM: P' x (AP) (implicit) (sub, total)	19.4298	19.9245	0.975	20.072	0.968	20.6297	0.942
MueLu: RAPFactory: MxM: P' x (AP) (implicit) (sub, total, level=1)	14.9438	15.0743	0.991	15.1627	0.986	15.5504	0.961
MueLu: RAPFactory: MxM: P' x (AP) (implicit) (sub, total, level=2)	3.64709	3.87942	0.94	3.8261	0.953	3.96852	0.919
MueLu: RAPFactory: Projections (sub, total)	0.553197	0.537245	1.03	0.53298	1.038	0.54963	1.006
MueLu: RAPFactory: Projections (sub, total, level=1)	0.459088	0.442652	1.037	0.440828	1.041	0.457331	1.004
MueLu: RAPFactory: Projections (sub, total, level=2)	0.0437806	0.0441141	0.992	0.0427427	1.024	0.0421915	1.038
MueLu: RebalanceAcFactory: Computing Ac	4.38985	4.37214	1.004	4.23797	1.036	4.30996	1.019
MueLu: RebalanceAcFactory: Computing Ac (level=1)	0.00212431	0.00143313	1.482	0.00126243	1.683	0.00118637	1.791
MueLu: RebalanceAcFactory: Computing Ac (level=2)	1.93823	1.94172	0.998	1.90854	1.016	1.93936	0.999
MueLu: RebalanceAcFactory: Computing Ac (total)	4.39406	4.37622	1.004	4.2421	1.036	4.31402	1.019
MueLu: RebalanceAcFactory: Computing Ac (total, level=1)	0.0028801	0.00229645	1.254	0.0019865	1.45	0.00196648	1.465
MueLu: RebalanceAcFactory: Computing Ac (total, level=2)	1.93938	1.94278	0.998	1.90959	1.016	1.94042	0.999
MueLu: RebalanceAcFactory: Rebalancing existing Ac (sub, total)	4.03935	3.9961	1.011	3.86374	1.045	3.9913	1.012
MueLu: RebalanceAcFactory: Rebalancing existing Ac (sub, total, level=2)	1.71885	1.68949	1.017	1.65434	1.039	1.74077	0.987
MueLu: RebalanceTransferFactory: Build	3.87917	1.52616	2.542	1.3145	2.951	1.33944	2.896
MueLu: RebalanceTransferFactory: Build (level=1)	0.0107551	0.0113413	0.948	0.0102029	1.054	0.0104659	1.028
MueLu: RebalanceTransferFactory: Build (level=2)	2.07421	0.817761	2.536	0.724756	2.862	0.720231	2.88
MueLu: RebalanceTransferFactory: Build (total)	113.462	91.9387	1.234	88.6467	1.28	90.1964	1.258
MueLu: RebalanceTransferFactory: Build (total, level=1)	62.4371	51.1897	1.22	49.7841	1.254	49.9544	1.25
MueLu: RebalanceTransferFactory: Build (total, level=2)	30.853	29.5515	1.044	27.9355	1.104	28.6459	1.077
MueLu: RebalanceTransferFactory: Rebalancing coordinates (sub, total)	3.08064	0.887559	3.471	0.763045	4.037	0.787341	3.913
MueLu: RebalanceTransferFactory: Rebalancing coordinates (sub, total, level=2)	1.58919	0.493579	3.22	0.429225	3.702	0.428761	3.706
MueLu: RebalanceTransferFactory: Rebalancing nullspace (sub, total)	0.796441	0.599047	1.33	0.554996	1.435	0.550843	1.446
MueLu: RebalanceTransferFactory: Rebalancing nullspace (sub, total, level=2)	0.481271	0.348315	1.382	0.285657	1.685	0.280139	1.718
MueLu: RebalanceTransferFactory: Rebalancing prolongator (sub, total)	0.00480199	0.00460958	1.042	0.00466204	1.03	0.00459599	1.045
MueLu: RebalanceTransferFactory: Rebalancing prolongator (sub, total, level=1)	0.00035429	0.000989676	0.358	0.000756025	0.469	0.00086236	0.411
MueLu: RebalanceTransferFactory: Rebalancing prolongator (sub, total, level=2)	0.000442982	0.000373602	1.186	0.000416279	1.064	0.000386	1.148
MueLu: RepartitionFactory: Build	8.20273	5.0768	1.616	4.8784	1.681	4.85286	1.69
MueLu: RepartitionFactory: Build (level=1)	0.155378	0.151963	1.022	0.137552	1.13	0.135852	1.144
MueLu: RepartitionFactory: Build (level=2)	3.96878	2.846	1.395	2.73387	1.452	2.7429	1.447
MueLu: RepartitionFactory: Build (total)	109.585	90.4532	1.212	87.3644	1.254	88.8872	1.233
MueLu: RepartitionFactory: Build (total, level=1)	62.4274	51.1803	1.22	49.7747	1.254	49.945	1.25
MueLu: RepartitionFactory: Build (total, level=2)	28.787	28.7427	1.002	27.2206	1.058	27.9364	1.03
MueLu: RepartitionFactory: DeterminePartitionPlacement (sub, total)	5.09958	2.85444	1.787	2.72826	1.869	2.74443	1.858
MueLu: RepartitionFactory: DeterminePartitionPlacement (sub, total, level=2)	3.23265	2.00213	1.615	1.97182	1.639	1.94583	1.661
MueLu: RepartitionFactory: Import construction (sub, total)	0.451074	0.464675	0.971	0.477931	0.944	0.44793	1.007
MueLu: RepartitionFactory: Import construction (sub, total, level=2)	0.18352	0.188859	0.972	0.180045	1.019	0.17671	1.039
MueLu: RepartitionHeuristicFactory: Build	0.422336	0.308879	1.367	0.245962	1.717	0.24304	1.738
MueLu: RepartitionHeuristicFactory: Build (level=1)	0.00242209	0.00304389	0.796	0.0020771	1.166	0.00207782	1.166
MueLu: RepartitionHeuristicFactory: Build (level=2)	0.190611	0.167431	1.138	0.106676	1.787	0.104257	1.828
MueLu: RepartitionHeuristicFactory: Build (total)	0.427832	0.314034	1.362	0.251038	1.704	0.248243	1.723
MueLu: RepartitionHeuristicFactory: Build (total, level=1)	0.00385022	0.00426507	0.903	0.00330114	1.166	0.00339341	1.135
MueLu: RepartitionHeuristicFactory: Build (total, level=2)	0.19187	0.168721	1.137	0.108016	1.776	0.105598	1.817
MueLu: SaPFactory: Eigenvalue estimate (sub, total)	0.00747156	0.00752687	0.993	0.0071559	1.044	0.00726414	1.029
MueLu: SaPFactory: Eigenvalue estimate (sub, total, level=1)	0.00106382	0.0017314	0.614	0.00101924	1.044	0.00102043	1.043
MueLu: SaPFactory: Eigenvalue estimate (sub, total, level=2)	0.00110984	0.00114369	0.97	0.0010047	1.105	0.00105667	1.05
MueLu: SaPFactory: Fused (I-omegaD^{-1} A)Ptent (sub, total)	19.9095	18.5396	1.074	18.0151	1.105	17.6495	1.128
MueLu: SaPFactory: Fused (I-omegaD^{-1} A)Ptent (sub, total, level=1)	14.9151	14.7337	1.012	14.1598	1.053	13.6577	1.092
MueLu: SaPFactory: Fused (I-omegaD^{-1} A)Ptent (sub, total, level=2)	2.76095	2.78643	0.991	2.80174	0.985	2.88711	0.956
MueLu: SaPFactory: Prolongator smoothing	21.3413	19.5351	1.092	18.8414	1.133	18.5431	1.151
MueLu: SaPFactory: Prolongator smoothing (level=1)	15.3674	14.9902	1.025	14.418	1.066	13.9242	1.104
MueLu: SaPFactory: Prolongator smoothing (level=2)	2.97511	3.07146	0.969	2.99497	0.993	3.12732	0.951
MueLu: SaPFactory: Prolongator smoothing (total)	28.6188	27.3451	1.047	25.9417	1.103	25.5798	1.119
MueLu: SaPFactory: Prolongator smoothing (total, level=1)	20.1302	20.1067	1.001	19.0078	1.059	18.4813	1.089
MueLu: SaPFactory: Prolongator smoothing (total, level=2)	4.51119	4.71515	0.957	4.59568	0.982	4.70883	0.958
MueLu: TentativePFactory: Build	2.40339	2.49933	0.962	2.27357	1.057	2.27681	1.056
MueLu: TentativePFactory: Build (level=1)	1.76949	1.89813	0.932	1.69149	1.046	1.67733	1.055
MueLu: TentativePFactory: Build (level=2)	0.311484	0.326388	0.954	0.299991	1.038	0.330996	0.941
MueLu: TentativePFactory: Build (total)	5.96015	6.30461	0.945	5.70099	1.045	5.68293	1.049
MueLu: TentativePFactory: Build (total, level=1)	3.79277	4.02249	0.943	3.62576	1.046	3.58911	1.057
MueLu: TentativePFactory: Build (total, level=2)	1.31374	1.35348	0.971	1.27898	1.027	1.31147	1.002
MueLu: UncoupledAggregationFactory: Algo "Phase - (Dirichlet)" (sub, total)	0.0660655	0.0650613	1.015	0.062103	1.064	0.0633287	1.043
MueLu: UncoupledAggregationFactory: Algo "Phase - (Dirichlet)" (sub, total, level=0)	0.0268703	0.0274632	0.978	0.0253232	1.061	0.0253494	1.06
MueLu: UncoupledAggregationFactory: Algo "Phase - (Dirichlet)" (sub, total, level=1)	0.0147123	0.0126295	1.165	0.0128918	1.141	0.0138102	1.065
MueLu: UncoupledAggregationFactory: Algo "Phase 1 (main)" (sub, total)	1.36626	1.4652	0.932	1.35065	1.012	1.35934	1.005
MueLu: UncoupledAggregationFactory: Algo "Phase 1 (main)" (sub, total, level=0)	0.894507	0.963669	0.928	0.878775	1.018	0.886593	1.009
MueLu: UncoupledAggregationFactory: Algo "Phase 1 (main)" (sub, total, level=1)	0.357106	0.386757	0.923	0.359107	0.994	0.359862	0.992
MueLu: UncoupledAggregationFactory: Algo "Phase 2a (secondary)" (sub, total)	0.528727	0.541357	0.977	0.525506	1.006	0.523707	1.01
MueLu: UncoupledAggregationFactory: Algo "Phase 2a (secondary)" (sub, total, level=0)	0.239567	0.246157	0.973	0.24135	0.993	0.241149	0.993
MueLu: UncoupledAggregationFactory: Algo "Phase 2a (secondary)" (sub, total, level=1)	0.194484	0.200179	0.972	0.191296	1.017	0.190155	1.023
MueLu: UncoupledAggregationFactory: Algo "Phase 2b (expansion)" (sub, total)	1.22668	1.2428	0.987	1.15667	1.061	1.12355	1.092
MueLu: UncoupledAggregationFactory: Algo "Phase 2b (expansion)" (sub, total, level=0)	0.726562	0.732728	0.992	0.662261	1.097	0.635218	1.144
MueLu: UncoupledAggregationFactory: Algo "Phase 2b (expansion)" (sub, total, level=1)	0.354959	0.361943	0.981	0.352059	1.008	0.348195	1.019
MueLu: UncoupledAggregationFactory: Algo "Phase 3 (cleanup)" (sub, total)	0.0793405	0.0711732	1.115	0.0712705	1.113	0.0725293	1.094
MueLu: UncoupledAggregationFactory: Algo "Phase 3 (cleanup)" (sub, total, level=0)	0.0371068	0.0336924	1.101	0.034344	1.08	0.0372431	0.996
MueLu: UncoupledAggregationFactory: Algo "Phase 3 (cleanup)" (sub, total, level=1)	0.0209806	0.0129275	1.623	0.0137093	1.53	0.0163782	1.281
MueLu: UncoupledAggregationFactory: Build	3.43964	3.549	0.969	3.32657	1.034	3.30651	1.04
MueLu: UncoupledAggregationFactory: Build (level=0)	1.99522	2.07631	0.961	1.91172	1.044	1.88991	1.056
MueLu: UncoupledAggregationFactory: Build (level=1)	0.975313	1.00467	0.971	0.959612	1.016	0.959644	1.016
MueLu: UncoupledAggregationFactory: Build (total)	3.44349	3.55272	0.969	3.33032	1.034	3.31039	1.04
MueLu: UncoupledAggregationFactory: Build (total, level=0)	1.99629	2.0774	0.961	1.91278	1.044	1.89098	1.056
MueLu: UncoupledAggregationFactory: Build (total, level=1)	0.976387	1.00569	0.971	0.960633	1.016	0.960672	1.016
MueLu: Zoltan2Interface: Build	17.3547	13.9386	1.245	12.5566	1.382	13.5379	1.282
MueLu: Zoltan2Interface: Build (level=1)	0.00221205	0.00267506	0.827	0.0022428	0.986	0.0022223	0.995
MueLu: Zoltan2Interface: Build (level=2)	11.8378	12.4827	0.948	11.1326	1.063	11.5518	1.025
MueLu: Zoltan2Interface: Build (total)	17.3581	13.9428	1.245	12.5606	1.382	13.542	1.282
MueLu: Zoltan2Interface: Build (total, level=1)	0.00314283	0.00356054	0.883	0.00308585	1.018	0.00313807	1.002
MueLu: Zoltan2Interface: Build (total, level=2)	11.8391	12.4839	0.948	11.1337	1.063	11.5529	1.025
MueLu: Zoltan2Interface: Zoltan2 multijagged (sub, total)	16.8158	13.5781	1.238	12.1842	1.38	13.1849	1.275
MueLu: Zoltan2Interface: Zoltan2 multijagged (sub, total, level=2)	11.6026	12.3213	0.942	10.9806	1.057	11.4033	1.017
MultiJagged - Final DistributorPlanComm		0.243017		0.0908208		0.234641
MultiJagged - Final DistributorPlanCreating		0.676229		0.492985		0.764104
MultiJagged - Migration DistributorMigration-0		0.135608		0.133178		0.130844
MultiJagged - Migration DistributorMigration-1		0.151176		0.097888		0.109345
MultiJagged - Migration DistributorPlanCreating-0		1.47905		0.731299		0.836204
MultiJagged - Migration DistributorPlanCreating-1		0.172276		0.0601864		0.0962286
MultiJagged - Part_Assignment		0.772875		0.555005		0.866364
MultiJagged - PostMigration DistributorMigration		0.0456626		0.0574806		0.0547552
MultiJagged - PostMigration DistributorPlanCreating		12.6652		11.3486		12.2846
MultiJagged - PreMigration DistributorMigration		0.0691514		0.076077		0.0754247
MultiJagged - PreMigration DistributorPlanCreating		0.687274		0.463595		0.533804
MultiJagged - Problem_Free		0.000704765		0.000634909		0.000650406
MultiJagged - Problem_Migration-0		9.75678		9.27724		9.77918
MultiJagged - Problem_Migration-1		0.704301		0.620316		0.641442
MultiJagged - Problem_Partitioning		11.5054		10.5979		11.1695
MultiJagged - Problem_Partitioning_0		9.98574		9.548		10.0623
MultiJagged - Problem_Partitioning_1		1.27529		0.875352		0.924565
MultiJagged - Problem_Partitioning_2		0.239334		0.124609		0.144092
MultiJagged - Solution_Part_Assignment		0.000112772		9.87053e-05		0.00011611
MultiJagged - Total		12.2412		10.9856		11.921
Tpetra MueLu::A*P-1: ESFC-G-Maps	0.000213146	0.000234365	0.909	0.000219822	0.97	0.00022006	0.969
Tpetra MueLu::A*P-1: ESFC-G-Setup	0.000243425	0.000249386	0.976	0.000253916	0.959	0.000241518	1.008
Tpetra MueLu::A*P-1: ESFC-G-cGC (noconst)	0.960211	0.868415	1.106	0.893727	1.074	0.953424	1.007
Tpetra MueLu::A*P-1: ESFC-G-cIS	0.000133038	0.000148058	0.899	0.000116587	1.141	0.000156879	0.848
Tpetra MueLu::A*P-1: ESFC-G-fLG	0.00102472	0.000824213	1.243	0.000866413	1.183	0.000859976	1.192
Tpetra MueLu::A*P-1: ESFC-G-mIXcheckE	0.000156641	0.00018549	0.844	0.000187397	0.836	0.000189781	0.825
Tpetra MueLu::A*P-1: ESFC-G-mIXcheckI	0.00024581	0.00029707	0.827	0.000286102	0.859	0.000252008	0.975
Tpetra MueLu::A*P-1: ESFC-G-mIXmake	14.4907	2.86198	5.063	2.42787	5.968	2.43163	5.959
Tpetra MueLu::A*P-1: ESFC-M-Graph	15.2329	3.62227	4.205	3.15154	4.833	3.15501	4.828
Tpetra MueLu::A*P-1: ESFC-M-cGC	0.000280857	0.000289202	0.971	0.000267267	1.051	0.000264883	1.06
Tpetra MueLu::A*P-1: ESFC-M-cIS	0.000115871	0.000125647	0.922	0.000144243	0.803	0.000127077	0.912
Tpetra MueLu::A*P-1: ESFC-M-fLGAM	0.000522852	0.00058198	0.898	0.000588417	0.889	0.0006001	0.871
Tpetra MueLu::A*P-1: TAFC CreateImporter	0.789808	0.867495	0.91	0.960713	0.822	1.10737	0.713
Tpetra MueLu::A*P-1: TAFC ESFC	0.0941415	0.0901935	1.044	0.0905366	1.04	0.0902789	1.043
Tpetra MueLu::A*P-1: TAFC ImportSetup	0.00257492	0.0035305	0.729	0.00261068	0.986	0.00249505	1.032
Tpetra MueLu::A*P-1: TAFC Pack-1	0.0848856	0.0870714	0.975	0.0835686	1.016	0.0829051	1.024
Tpetra MueLu::A*P-1: TAFC Pack-2	0.492177	0.491426	1.002	0.490884	1.003	0.491091	1.002
Tpetra MueLu::A*P-1: TAFC Transfer	0.632481	0.632688	1.0	0.779472	0.811	0.899404	0.703
Tpetra MueLu::A*P-1: TAFC Unpack-1	0.208941	0.206635	1.011	0.207234	1.008	0.2072	1.008
Tpetra MueLu::A*P-1: TAFC Unpack-2	1.24571	1.23866	1.006	1.35878	0.917	1.44681	0.861
Tpetra MueLu::A*P-1: TAFC Unpack-3	0.0701952	0.069685	1.007	0.0699682	1.003	0.0700932	1.001
Tpetra MueLu::A*P-2: ESFC-G-Maps	0.000231743	0.000257492	0.9	0.000268936	0.862	0.000242472	0.956
Tpetra MueLu::A*P-2: ESFC-G-Setup	0.00026536	0.000294685	0.9	0.000291348	0.911	0.000276089	0.961
Tpetra MueLu::A*P-2: ESFC-G-cGC (noconst)	0.0926819	0.0975437	0.95	0.0927556	0.999	0.0930638	0.996
Tpetra MueLu::A*P-2: ESFC-G-cIS	0.00015831	0.000152349	1.039	0.000150919	1.049	0.000121355	1.305
Tpetra MueLu::A*P-2: ESFC-G-fLG	0.00120544	0.00109696	1.099	0.00114441	1.053	0.00117636	1.025
Tpetra MueLu::A*P-2: ESFC-G-mIXcheckE	0.000192165	0.000199318	0.964	0.000198126	0.97	0.000196218	0.979
Tpetra MueLu::A*P-2: ESFC-G-mIXcheckI	0.000322104	0.000319719	1.007	0.00032568	0.989	0.000294685	1.093
Tpetra MueLu::A*P-2: ESFC-G-mIXmake	0.000365973	0.000405312	0.903	0.000382423	0.957	0.000409842	0.893
Tpetra MueLu::A*P-2: ESFC-M-Graph	0.0942914	0.0992177	0.95	0.0943978	0.999	0.0950406	0.992
Tpetra MueLu::A*P-2: ESFC-M-cGC	0.000284433	0.000268459	1.06	0.000326872	0.87	0.000287533	0.989
Tpetra MueLu::A*P-2: ESFC-M-cIS	0.000103951	0.00014472	0.718	0.000158072	0.658	0.00014019	0.742
Tpetra MueLu::A*P-2: ESFC-M-fLGAM	0.000676632	0.000695229	0.973	0.000746727	0.906	0.000702858	0.963
Tpetra MueLu::A*P-2: TAFC CreateImporter	0.250885	0.328168	0.765	0.30108	0.833	0.316898	0.792
Tpetra MueLu::A*P-2: TAFC ESFC	0.0409629	0.0409813	1.0	0.040772	1.005	0.040803	1.004
Tpetra MueLu::A*P-2: TAFC ImportSetup	0.000793695	0.000754118	1.052	0.000813246	0.976	0.000899553	0.882
Tpetra MueLu::A*P-2: TAFC Pack-1	0.023417	0.0137889	1.698	0.0133944	1.748	0.0131536	1.78
Tpetra MueLu::A*P-2: TAFC Pack-2	0.215563	0.21398	1.007	0.214559	1.005	0.215198	1.002
Tpetra MueLu::A*P-2: TAFC Transfer	0.67341	0.592407	1.137	0.75954	0.887	0.841502	0.8
Tpetra MueLu::A*P-2: TAFC Unpack-1	0.133682	0.128424	1.041	0.128049	1.044	0.128552	1.04
Tpetra MueLu::A*P-2: TAFC Unpack-2	0.832224	0.754704	1.103	0.965819	0.862	1.07507	0.774
Tpetra MueLu::A*P-2: TAFC Unpack-3	0.0693016	0.0688767	1.006	0.0688522	1.007	0.0688725	1.006
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-Maps	0.000206232	0.000230312	0.895	0.000228167	0.904	0.000223875	0.921
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-Setup	0.000360727	0.000376463	0.958	0.000339031	1.064	0.000339746	1.062
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-cGC (const)	0.116662	0.124656	0.936	0.11658	1.001	0.112907	1.033
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-cIS	0.000114918	0.00014782	0.777	0.000123739	0.929	0.000121593	0.945
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-fLG	0.000973225	0.000961065	1.013	0.000927925	1.049	0.000925064	1.052
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-mIXcheckE	0.0001719	0.00018692	0.92	0.000188112	0.914	0.000185251	0.928
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-mIXcheckI	0.000276327	0.000297546	0.929	0.000273466	1.01	0.000275612	1.003
Tpetra MueLu::R*(AP)-implicit-1: ESFC-G-mIXmake	1.95565	2.35381	0.831	2.10663	0.928	2.23185	0.876
Tpetra MueLu::R*(AP)-implicit-1: ESFC-M-Graph	2.06914	2.4755	0.836	2.21376	0.935	2.34068	0.884
Tpetra MueLu::R*(AP)-implicit-1: ESFC-M-cGC	0.000263691	0.000270128	0.976	0.000273228	0.965	0.000270367	0.975
Tpetra MueLu::R*(AP)-implicit-1: ESFC-M-cIS	0.000127077	0.000111818	1.136	0.000107765	1.179	0.000137806	0.922
Tpetra MueLu::R*(AP)-implicit-1: ESFC-M-fLGAM	0.000524282	0.000577688	0.908	0.000571966	0.917	0.000612736	0.856
Tpetra MueLu::R*(AP)-implicit-1: TAFC CreateImporter	1.86629	1.51341	1.233	1.76362	1.058	1.90343	0.98
Tpetra MueLu::R*(AP)-implicit-1: TAFC ESFC	0.0932133	0.0978882	0.952	0.0849831	1.097	0.0896685	1.04
Tpetra MueLu::R*(AP)-implicit-1: TAFC ImportSetup	0.00272679	0.00265741	1.026	0.00261593	1.042	0.00274944	0.992
Tpetra MueLu::R*(AP)-implicit-1: TAFC Pack-1	0.00134444	0.00245619	0.547	0.00129461	1.038	0.00138712	0.969
Tpetra MueLu::R*(AP)-implicit-1: TAFC Pack-2	0.146011	0.145165	1.006	0.149838	0.974	0.145386	1.004
Tpetra MueLu::R*(AP)-implicit-1: TAFC Transfer	0.785847	0.622534	1.262	0.760515	1.033	0.858794	0.915
Tpetra MueLu::R*(AP)-implicit-1: TAFC Unpack-1	1.39933	1.37527	1.017	1.44003	0.972	1.42818	0.98
Tpetra MueLu::R*(AP)-implicit-1: TAFC Unpack-2	1.94843	1.81999	1.071	2.01532	0.967	2.13472	0.913
Tpetra MueLu::R*(AP)-implicit-1: TAFC Unpack-3	1.53542	1.22059	1.258	1.20365	1.276	1.19923	1.28
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-Maps	0.000359535	0.000340223	1.057	0.000350475	1.026	0.00035429	1.015
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-Setup	0.000436544	0.000410318	1.064	0.000420809	1.037	0.00044322	0.985
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-cGC (noconst)	0.137593	0.131044	1.05	0.132261	1.04	0.132231	1.041
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-cIS	0.00012517	0.00015974	0.784	0.000154257	0.811	0.000156164	0.802
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-fLG	0.000835657	0.000843525	0.991	0.000896215	0.932	0.00086236	0.969
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-mIXcheckE	0.000246525	0.000272989	0.903	0.000262976	0.937	0.000264883	0.931
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-mIXcheckI	0.000191212	0.000206947	0.924	0.000204325	0.936	0.00018549	1.031
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-G-mIXmake	0.38445	0.519726	0.74	0.507006	0.758	0.539052	0.713
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-M-Graph	0.494341	0.628767	0.786	0.616533	0.802	0.642412	0.77
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-M-cGC	0.000270367	0.000268698	1.006	0.000258207	1.047	0.000289917	0.933
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-M-cIS	9.29832e-05	0.000139713	0.666	0.000139713	0.666	0.00014472	0.643
Tpetra MueLu::R*(AP)-implicit-1XP: : ESFC-M-fLGAM	0.000508785	0.000554323	0.918	0.000562668	0.904	0.000594854	0.855
Tpetra MueLu::R*(AP)-implicit-1XP: : Transpose Local	1.08264	1.20772	0.896	1.18108	0.917	1.2107	0.894
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-Maps	0.000210047	0.000217199	0.967	0.000213146	0.985	0.000219822	0.956
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-Setup	0.000225782	0.000232935	0.969	0.000245094	0.921	0.000226974	0.995
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-cGC (const)	0.0366697	0.030709	1.194	0.0370729	0.989	0.0452993	0.809
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-cIS	0.000118256	0.000177145	0.668	0.000184774	0.64	0.000174761	0.677
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-fLG	0.00107765	0.00104618	1.03	0.00103807	1.038	0.00106311	1.014
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-mIXcheckE	0.000177383	0.00019598	0.905	0.000189304	0.937	0.000191689	0.925
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-mIXcheckI	0.000239611	0.000250578	0.956	0.000249624	0.96	0.000259399	0.924
Tpetra MueLu::R*(AP)-implicit-2: ESFC-G-mIXmake	1.18738	1.364	0.871	1.1939	0.995	1.22489	0.969
Tpetra MueLu::R*(AP)-implicit-2: ESFC-M-Graph	1.22339	1.393	0.878	1.22533	0.998	1.27041	0.963
Tpetra MueLu::R*(AP)-implicit-2: ESFC-M-cGC	0.000290632	0.000235796	1.233	0.000286341	1.015	0.000283003	1.027
Tpetra MueLu::R*(AP)-implicit-2: ESFC-M-cIS	0.000142813	0.000117064	1.22	0.000132799	1.075	0.000158072	0.903
Tpetra MueLu::R*(AP)-implicit-2: ESFC-M-fLGAM	0.000560045	0.00060463	0.926	0.000604391	0.927	0.000608206	0.921
Tpetra MueLu::R*(AP)-implicit-2: TAFC CreateImporter	0.363373	0.466873	0.778	0.436021	0.833	0.414962	0.876
Tpetra MueLu::R*(AP)-implicit-2: TAFC ESFC	0.0246754	0.0263977	0.935	0.0223608	1.104	0.0212433	1.162
Tpetra MueLu::R*(AP)-implicit-2: TAFC ImportSetup	0.000648022	0.000612736	1.058	0.000656128	0.988	0.000604391	1.072
Tpetra MueLu::R*(AP)-implicit-2: TAFC Pack-1	0.00139928	0.0013628	1.027	0.00137949	1.014	0.00135469	1.033
Tpetra MueLu::R*(AP)-implicit-2: TAFC Pack-2	0.0376947	0.0368948	1.022	0.0377455	0.999	0.0379379	0.994
Tpetra MueLu::R*(AP)-implicit-2: TAFC Transfer	0.486998	0.411558	1.183	0.550584	0.885	0.642483	0.758
Tpetra MueLu::R*(AP)-implicit-2: TAFC Unpack-1	0.099299	0.0989268	1.004	0.0991187	1.002	0.0990212	1.003
Tpetra MueLu::R*(AP)-implicit-2: TAFC Unpack-2	0.660525	0.607981	1.086	0.768182	0.86	0.894354	0.739
Tpetra MueLu::R*(AP)-implicit-2: TAFC Unpack-3	0.155469	0.155531	1.0	0.15578	0.998	0.155844	0.998
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-Maps	0.000308514	0.000274181	1.125	0.000284433	1.085	0.000279427	1.104
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-Setup	0.000306368	0.00029707	1.031	0.000321388	0.953	0.000313997	0.976
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-cGC (noconst)	0.00740433	0.00734735	1.008	0.00753236	0.983	0.00744677	0.994
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-cIS	0.000166416	0.00016737	0.994	0.00016427	1.013	0.000159264	1.045
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-fLG	0.000886917	0.000834703	1.063	0.000841856	1.054	0.000813961	1.09
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-mIXcheckE	0.000210762	0.000247955	0.85	0.000231028	0.912	0.000264883	0.796
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-mIXcheckI	0.000193119	0.000199556	0.968	0.000196457	0.983	0.000196218	0.984
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-G-mIXmake	0.000333071	0.000339746	0.98	0.000379801	0.877	0.000347376	0.959
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-M-Graph	0.00949812	0.00931406	1.02	0.00957966	0.991	0.00943971	1.006
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-M-cGC	0.00027442	0.000281096	0.976	0.000279665	0.981	0.000277758	0.988
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-M-cIS	0.000133276	0.000153065	0.871	0.000145674	0.915	0.000168562	0.791
Tpetra MueLu::R*(AP)-implicit-2XP: : ESFC-M-fLGAM	0.000628471	0.000701904	0.895	0.000693798	0.906	0.000695467	0.904
Tpetra MueLu::R*(AP)-implicit-2XP: : Transpose Local	0.139919	0.13923	1.005	0.140324	0.997	0.140032	0.999
Tpetra MueLu::RebalanceAc-2: TAFC ImportSetup	0.18135	0.193561	0.937	0.175954	1.031	0.17592	1.031
Tpetra MueLu::RebalanceAc-2: TAFC Pack-1	0.473087	0.311772	1.517	0.285072	1.66	0.283419	1.669
Tpetra MueLu::RebalanceAc-2: TAFC Pack-2	0.0501549	0.0497396	1.008	0.0498197	1.007	0.049489	1.013
Tpetra MueLu::RebalanceAc-2: TAFC Transfer	0.533876	0.35668	1.497	0.469648	1.137	0.575601	0.928
Tpetra MueLu::RebalanceAc-2: TAFC Unpack-1	0.0787764	0.0777903	1.013	0.077939	1.011	0.0777304	1.013
Tpetra MueLu::RebalanceAc-2: TAFC Unpack-2	0.769776	0.724965	1.062	0.860646	0.894	0.96365	0.799
Tpetra MueLu::SaP-1: ESFC-G-Maps	0.000209332	0.000217676	0.962	0.000208139	1.006	0.00022006	0.951
Tpetra MueLu::SaP-1: ESFC-G-Setup	0.000236988	0.000244141	0.971	0.000247478	0.958	0.000242233	0.978
Tpetra MueLu::SaP-1: ESFC-G-cGC (noconst)	0.77675	0.771171	1.007	0.729377	1.065	0.722889	1.075
Tpetra MueLu::SaP-1: ESFC-G-cIS	0.000127792	0.000146389	0.873	0.000130892	0.976	0.000147581	0.866
Tpetra MueLu::SaP-1: ESFC-G-fLG	0.000777483	0.000774622	1.004	0.000814438	0.955	0.000810146	0.96
Tpetra MueLu::SaP-1: ESFC-G-mIXcheckE	0.00014782	0.000180721	0.818	0.000189543	0.78	0.000184059	0.803
Tpetra MueLu::SaP-1: ESFC-G-mIXcheckI	0.000259638	0.000264406	0.982	0.000266552	0.974	0.000268698	0.966
Tpetra MueLu::SaP-1: ESFC-G-mIXmake	2.21841	1.27873	1.735	0.987842	2.246	0.994603	2.23
Tpetra MueLu::SaP-1: ESFC-M-Graph	2.88767	1.94924	1.481	1.6498	1.75	1.6563	1.743
Tpetra MueLu::SaP-1: ESFC-M-cGC	0.00026083	0.000269651	0.967	0.000261307	0.998	0.000257492	1.013
Tpetra MueLu::SaP-1: ESFC-M-cIS	0.000132084	0.000142813	0.925	0.000131607	1.004	0.00013423	0.984
Tpetra MueLu::SaP-1: ESFC-M-fLGAM	0.000507593	0.000538111	0.943	0.000591278	0.858	0.000564337	0.899
Tpetra MueLu::SaP-1: TAFC CreateImporter	0.291214	0.305663	0.953	0.38498	0.756	0.304324	0.957
Tpetra MueLu::SaP-1: TAFC ESFC	0.0780396	0.0720658	1.083	0.0723801	1.078	0.0732408	1.066
Tpetra MueLu::SaP-1: TAFC ImportSetup	0.00150156	0.00224614	0.669	0.00135064	1.112	0.00137019	1.096
Tpetra MueLu::SaP-1: TAFC Pack-1	0.0872867	0.082608	1.057	0.105134	0.83	0.0808802	1.079
Tpetra MueLu::SaP-1: TAFC Pack-2	0.40783	0.409325	0.996	0.410129	0.994	0.406799	1.003
Tpetra MueLu::SaP-1: TAFC Transfer	0.297761	0.315326	0.944	0.573803	0.519	0.328506	0.906
Tpetra MueLu::SaP-1: TAFC Unpack-1	0.172205	0.181786	0.947	0.182669	0.943	0.181659	0.948
Tpetra MueLu::SaP-1: TAFC Unpack-2	0.504829	0.550505	0.917	0.862509	0.585	0.517375	0.976
Tpetra MueLu::SaP-1: TAFC Unpack-3	0.0144041	0.0151258	0.952	0.0144513	0.997	0.0144489	0.997
Tpetra MueLu::SaP-2: ESFC-G-Maps	0.000255823	0.000252962	1.011	0.000282288	0.906	0.00025773	0.993
Tpetra MueLu::SaP-2: ESFC-G-Setup	0.000279188	0.000281334	0.992	0.00028491	0.98	0.000252485	1.106
Tpetra MueLu::SaP-2: ESFC-G-cGC (noconst)	0.0843124	0.0900221	0.937	0.0845048	0.998	0.084394	0.999
Tpetra MueLu::SaP-2: ESFC-G-cIS	0.000161409	0.000157118	1.027	0.000167847	0.962	0.000164509	0.981
Tpetra MueLu::SaP-2: ESFC-G-fLG	0.000764608	0.000778913	0.982	0.000788212	0.97	0.000795603	0.961
Tpetra MueLu::SaP-2: ESFC-G-mIXcheckE	0.000183582	0.000196457	0.934	0.00019598	0.937	0.000200272	0.917
Tpetra MueLu::SaP-2: ESFC-G-mIXcheckI	0.000291824	0.000300407	0.971	0.000296831	0.983	0.000290394	1.005
Tpetra MueLu::SaP-2: ESFC-G-mIXmake	0.00036931	0.000395298	0.934	0.000379086	0.974	0.000400305	0.923
Tpetra MueLu::SaP-2: ESFC-M-Graph	0.0859585	0.0917017	0.937	0.0861876	0.997	0.085989	1.0
Tpetra MueLu::SaP-2: ESFC-M-cGC	0.000282526	0.000340939	0.829	0.000282764	0.999	0.000295639	0.956
Tpetra MueLu::SaP-2: ESFC-M-cIS	0.000103474	0.000162601	0.636	0.000168562	0.614	0.000158787	0.652
Tpetra MueLu::SaP-2: ESFC-M-fLGAM	0.000535727	0.000586033	0.914	0.000594854	0.901	0.000622749	0.86
Tpetra MueLu::SaP-2: TAFC CreateImporter	0.223063	0.235137	0.949	0.167391	1.333	0.225296	0.99
Tpetra MueLu::SaP-2: TAFC ESFC	0.0373776	0.0371993	1.005	0.037503	0.997	0.0371852	1.005
Tpetra MueLu::SaP-2: TAFC ImportSetup	0.000574589	0.000597477	0.962	0.000579596	0.991	0.00056982	1.008
Tpetra MueLu::SaP-2: TAFC Pack-1	0.0111048	0.0112634	0.986	0.0111666	0.994	0.0111535	0.996
Tpetra MueLu::SaP-2: TAFC Pack-2	0.149453	0.148551	1.006	0.148523	1.006	0.148588	1.006
Tpetra MueLu::SaP-2: TAFC Transfer	0.41082	0.332295	1.236	0.484777	0.847	0.469966	0.874
Tpetra MueLu::SaP-2: TAFC Unpack-1	0.0802341	0.077929	1.03	0.0779171	1.03	0.0818427	0.98
Tpetra MueLu::SaP-2: TAFC Unpack-2	0.571327	0.494982	1.154	0.637686	0.896	0.651489	0.877
Tpetra MueLu::SaP-2: TAFC Unpack-3	0.00776553	0.00774622	1.002	0.00777936	0.998	0.00778055	0.998
Tpetra MueLu::TentativeP-1: ESFC-G-Maps	0.000338078	0.000314713	1.074	0.000302792	1.117	0.000323772	1.044
Tpetra MueLu::TentativeP-1: ESFC-G-Setup	0.000376701	0.000455618	0.827	0.00036025	1.046	0.000360012	1.046
Tpetra MueLu::TentativeP-1: ESFC-G-cGC (noconst)	0.419787	0.474076	0.885	0.420023	0.999	0.419611	1.0
Tpetra MueLu::TentativeP-1: ESFC-G-cIS	0.000141859	0.000173092	0.82	0.00016737	0.848	0.000130892	1.084
Tpetra MueLu::TentativeP-1: ESFC-G-fLG	0.000849247	0.00180912	0.469	0.000866652	0.98	0.000846148	1.004
Tpetra MueLu::TentativeP-1: ESFC-G-mIXcheckE	0.000176668	0.00024581	0.719	0.000201702	0.876	0.000198126	0.892
Tpetra MueLu::TentativeP-1: ESFC-G-mIXcheckI	0.000201225	0.000205994	0.977	0.000206232	0.976	0.000216246	0.931
Tpetra MueLu::TentativeP-1: ESFC-G-mIXmake	0.40488	0.442975	0.914	0.31864	1.271	0.334013	1.212
Tpetra MueLu::TentativeP-1: ESFC-M-Graph	0.800429	0.839226	0.954	0.713771	1.121	0.729053	1.098
Tpetra MueLu::TentativeP-1: ESFC-M-cGC	0.000254631	0.000250578	1.016	0.000253916	1.003	0.000243664	1.045
Tpetra MueLu::TentativeP-1: ESFC-M-cIS	0.00013423	0.000152826	0.878	0.000142813	0.94	0.000108719	1.235
Tpetra MueLu::TentativeP-1: ESFC-M-fLGAM	0.000552654	0.00059104	0.935	0.000585318	0.944	0.000588179	0.94
Tpetra MueLu::TentativeP-2: ESFC-G-Maps	0.000286102	0.000286341	0.999	0.00028944	0.988	0.000325203	0.88
Tpetra MueLu::TentativeP-2: ESFC-G-Setup	0.000373363	0.000294447	1.268	0.000292063	1.278	0.000328064	1.138
Tpetra MueLu::TentativeP-2: ESFC-G-cGC (noconst)	0.0523033	0.0597982	0.875	0.0524547	0.997	0.0527885	0.991
Tpetra MueLu::TentativeP-2: ESFC-G-cIS	0.000166416	0.000167847	0.991	0.00016427	1.013	0.00012517	1.33
Tpetra MueLu::TentativeP-2: ESFC-G-fLG	0.000638962	0.000626326	1.02	0.000683069	0.935	0.000638485	1.001
Tpetra MueLu::TentativeP-2: ESFC-G-mIXcheckE	0.000178814	0.000205278	0.871	0.000217915	0.821	0.000217199	0.823
Tpetra MueLu::TentativeP-2: ESFC-G-mIXcheckI	0.000208139	0.000211954	0.982	0.000222206	0.937	0.000211954	0.982
Tpetra MueLu::TentativeP-2: ESFC-G-mIXmake	0.000345707	0.000344515	1.003	0.000407934	0.847	0.000353575	0.978
Tpetra MueLu::TentativeP-2: ESFC-M-Graph	0.0538995	0.0612471	0.88	0.0538697	1.001	0.0541887	0.995
Tpetra MueLu::TentativeP-2: ESFC-M-cGC	0.000264645	0.000265121	0.998	0.000263214	1.005	0.000272512	0.971
Tpetra MueLu::TentativeP-2: ESFC-M-cIS	0.000142097	0.000122786	1.157	0.00011754	1.209	0.000170708	0.832
Tpetra MueLu::TentativeP-2: ESFC-M-fLGAM	0.000590563	0.000576973	1.024	0.000604868	0.976	0.000567913	1.04
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-Maps	0.000292301	0.000228882	1.277	0.000226974	1.288	0.000236034	1.238
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-Setup	0.000292778	0.000298023	0.982	0.000300646	0.974	0.000299692	0.977
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-cGC (noconst)	0.0900109	0.0862827	1.043	0.0863674	1.042	0.0866261	1.039
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-cIS	0.000183821	0.000174522	1.053	0.000183582	1.001	0.000183344	1.003
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-fLG	0.000893354	0.000926495	0.964	0.00091815	0.973	0.000892162	1.001
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-mIXcheckE	0.000160694	0.000177145	0.907	0.00016284	0.987	0.000184059	0.873
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-mIXcheckI	0.000254154	0.000264168	0.962	0.000257492	0.987	0.000258684	0.982
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-G-mIXmake	0.000327349	0.00032711	1.001	0.000374079	0.875	0.00032711	1.001
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-M-Graph	0.0917258	0.0879595	1.043	0.0881498	1.041	0.0881772	1.04
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-M-cGC	0.000302076	0.000324488	0.931	0.00033021	0.915	0.000313282	0.964
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-M-cIS	0.000178337	0.000169277	1.054	0.000158072	1.128	0.000167131	1.067
Tpetra Tpetra MueLu::A*P-1: TAFC: ESFC-M-fLGAM	0.000605822	0.000630617	0.961	0.000664473	0.912	0.000609875	0.993
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-Maps	0.000255585	0.000276804	0.923	0.000266075	0.961	0.000319481	0.8
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-Setup	0.000280619	0.00028491	0.985	0.000289679	0.969	0.000278711	1.007
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-cGC (noconst)	0.0369918	0.0371308	0.996	0.0369561	1.001	0.0369124	1.002
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-cIS	0.000192404	0.000196218	0.981	0.000203609	0.945	0.000192881	0.998
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-fLG	0.00088048	0.000832081	1.058	0.000830412	1.06	0.0008564	1.028
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-mIXcheckE	0.000181913	0.000204802	0.888	0.000215054	0.846	0.000194073	0.937
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-mIXcheckI	0.000289202	0.000278234	1.039	0.000276327	1.047	0.000282526	1.024
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-G-mIXmake	0.000393629	0.000355959	1.106	0.000361919	1.088	0.000350475	1.123
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-M-Graph	0.0386376	0.0387223	0.998	0.0384917	1.004	0.0384064	1.006
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-M-cGC	0.000314236	0.000316858	0.992	0.000307322	1.022	0.000320196	0.981
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-M-cIS	0.000168562	0.00018692	0.902	0.000189781	0.888	0.000185013	0.911
Tpetra Tpetra MueLu::A*P-2: TAFC: ESFC-M-fLGAM	0.000578642	0.000643253	0.9	0.000652075	0.887	0.000626564	0.924
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-Maps	0.000254393	0.000241518	1.053	0.000240088	1.06	0.000248432	1.024
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-Setup	0.000269413	0.000275135	0.979	0.000277758	0.97	0.000263691	1.022
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-cGC (const)	0.0889127	0.0939975	0.946	0.0809987	1.098	0.0857456	1.037
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-cIS	0.000131607	0.000152111	0.865	0.0001688	0.78	0.000114441	1.15
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-fLG	0.000967264	0.00101399	0.954	0.00101614	0.952	0.000989676	0.977
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-mIXcheckE	0.000215292	0.000198126	1.087	0.000181198	1.188	0.000201464	1.069
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-mIXcheckI	0.000263929	0.000265598	0.994	0.000261784	1.008	0.000263214	1.003
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-G-mIXmake	0.000370979	0.000359058	1.033	0.0003407	1.089	0.000335217	1.107
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-M-Graph	0.0908434	0.0957615	0.949	0.0828195	1.097	0.0875158	1.038
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-M-cGC	0.000283003	0.000293016	0.966	0.000319481	0.886	0.000275135	1.029
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-M-cIS	0.000118256	0.000150681	0.785	0.000155926	0.758	0.000107527	1.1
Tpetra Tpetra MueLu::R*(AP)-implicit-1: TAFC: ESFC-M-fLGAM	0.000515223	0.000577688	0.892	0.000594139	0.867	0.000585079	0.881
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-Maps	0.000253201	0.000240803	1.051	0.000234127	1.081	0.000244141	1.037
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-Setup	0.000303507	0.000286341	1.06	0.000266552	1.139	0.000286818	1.058
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-cGC (const)	0.0207694	0.0224321	0.926	0.0183876	1.13	0.0174017	1.194
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-cIS	0.000116587	0.000169754	0.687	0.000186682	0.625	0.000125408	0.93
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-fLG	0.00087142	0.00080657	1.08	0.000859499	1.014	0.000823975	1.058
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-mIXcheckE	0.00019598	0.000211239	0.928	0.000211239	0.928	0.00022006	0.891
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-mIXcheckI	0.000267982	0.000268936	0.996	0.000261068	1.026	0.000280857	0.954
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-G-mIXmake	0.000366688	0.000356197	1.029	0.000341654	1.073	0.000348568	1.052
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-M-Graph	0.0224004	0.0239978	0.933	0.0200133	1.119	0.0190101	1.178
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-M-cGC	0.000270605	0.000267982	1.01	0.000257015	1.053	0.000250578	1.08
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-M-cIS	0.000106573	0.000152111	0.701	0.000174046	0.612	0.000167131	0.638
Tpetra Tpetra MueLu::R*(AP)-implicit-2: TAFC: ESFC-M-fLGAM	0.000543833	0.000574827	0.946	0.000606537	0.897	0.000660896	0.823
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-Maps	0.000248194	0.000214338	1.158	0.000213623	1.162	0.000211716	1.172
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-Setup	0.000247955	0.000242472	1.023	0.000238419	1.04	0.000249624	0.993
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-cGC (noconst)	0.0739992	0.0686193	1.078	0.0688956	1.074	0.0694897	1.065
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-cIS	0.000192165	0.000211716	0.908	0.000183582	1.047	0.00018096	1.062
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-fLG	0.000861883	0.000850677	1.013	0.000892162	0.966	0.000912666	0.944
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-mIXcheckE	0.00017333	0.000169039	1.025	0.000172138	1.007	0.000155926	1.112
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-mIXcheckI	0.000243902	0.000232935	1.047	0.000230551	1.058	0.000220776	1.105
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-G-mIXmake	0.00030756	0.000329256	0.934	0.000298738	1.03	0.000302792	1.016
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-M-Graph	0.0757229	0.0702093	1.079	0.0705051	1.074	0.0710821	1.065
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-M-cGC	0.00036025	0.000346184	1.041	0.000336409	1.071	0.000350714	1.027
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-M-cIS	0.000159025	0.000183582	0.866	0.000169992	0.935	0.000158787	1.001
Tpetra Tpetra MueLu::SaP-1: TAFC: ESFC-M-fLGAM	0.000594139	0.000590324	1.006	0.000608444	0.976	0.000663519	0.895
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-Maps	0.000250816	0.000292778	0.857	0.000256777	0.977	0.000234365	1.07
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-Setup	0.000278234	0.000271797	1.024	0.000313044	0.889	0.000304222	0.915
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-cGC (noconst)	0.0337439	0.0336571	1.003	0.033885	0.996	0.0335748	1.005
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-cIS	0.000164747	0.000170708	0.965	0.000175476	0.939	0.000157595	1.045
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-fLG	0.00071764	0.000702381	1.022	0.000697374	1.029	0.000713348	1.006
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-mIXcheckE	0.000151396	0.000245571	0.617	0.000190735	0.794	0.000169516	0.893
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-mIXcheckI	0.000269413	0.00028348	0.95	0.000372171	0.724	0.000243664	1.106
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-G-mIXmake	0.000390768	0.000356197	1.097	0.000495911	0.788	0.000355005	1.101
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-M-Graph	0.0350904	0.0350988	1.0	0.0352926	0.994	0.0350091	1.002
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-M-cGC	0.000287056	0.000296116	0.969	0.000298262	0.962	0.000293493	0.978
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-M-cIS	0.000160217	0.000175953	0.911	0.000171185	0.936	0.000126362	1.268
Tpetra Tpetra MueLu::SaP-2: TAFC: ESFC-M-fLGAM	0.00055337	0.000591755	0.935	0.000631809	0.876	0.000606298	0.913
TpetraExt MueLu::A*P-1: MMM All I&X	3.17589	3.20826	0.99	3.41277	0.931	3.62924	0.875
TpetraExt MueLu::A*P-1: MMM All Multiply	23.469	12.0506	1.948	11.5538	2.031	11.5718	2.028
TpetraExt MueLu::A*P-1: MMM All Setup	0.000426769	0.000463486	0.921	0.000437498	0.975	0.000446558	0.956
TpetraExt MueLu::A*P-1: MMM I&X Alloc	0.000282049	0.000259399	1.087	0.000258684	1.09	0.000273943	1.03
TpetraExt MueLu::A*P-1: MMM I&X Collective-0	0.00799608	0.0151815	0.527	0.00684071	1.169	0.00513983	1.556
TpetraExt MueLu::A*P-1: MMM I&X Import-2	0.0760312	0.0758185	1.003	0.0779257	0.976	0.0749862	1.014
TpetraExt MueLu::A*P-1: MMM I&X Import-3	2.74706	2.823	0.973	3.04927	0.901	3.27686	0.838
TpetraExt MueLu::A*P-1: MMM I&X Import-4	0.000975132	0.000914574	1.066	0.000977039	0.998	0.00093627	1.042
TpetraExt MueLu::A*P-1: MMM I&X RemoteMap	0.356488	0.31248	1.141	0.287948	1.238	0.281529	1.266
TpetraExt MueLu::A*P-1: MMM M5 Cmap	23.4664	12.0479	1.948	11.5509	2.032	11.5692	2.028
TpetraExt MueLu::A*P-1: MMM Newmatrix ESFC	15.2355	3.65249	4.171	3.15406	4.83	3.15754	4.825
TpetraExt MueLu::A*P-1: MMM Newmatrix Final Sort	1.33762	1.33091	1.005	1.26799	1.055	1.28319	1.042
TpetraExt MueLu::A*P-1: MMM Newmatrix SerialCore	17.8523	6.32794	2.821	6.18222	2.888	6.09452	2.929
TpetraExt MueLu::A*P-2: MMM All I&X	1.75119	1.75287	0.999	1.90282	0.92	2.01152	0.871
TpetraExt MueLu::A*P-2: MMM All Multiply	3.12965	3.14346	0.996	3.1514	0.993	3.11226	1.006
TpetraExt MueLu::A*P-2: MMM All Setup	0.000376463	0.00041151	0.915	0.000422001	0.892	0.000398397	0.945
TpetraExt MueLu::A*P-2: MMM I&X Alloc	0.000290155	0.000294447	0.985	0.00031209	0.93	0.000312567	0.928
TpetraExt MueLu::A*P-2: MMM I&X Collective-0	0.00503421	0.00529385	0.951	0.00546312	0.921	0.00473738	1.063
TpetraExt MueLu::A*P-2: MMM I&X Import-2	0.0388782	0.0336258	1.156	0.035691	1.089	0.0373578	1.041
TpetraExt MueLu::A*P-2: MMM I&X Import-3	1.64919	1.66078	0.993	1.80996	0.911	1.91825	0.86
TpetraExt MueLu::A*P-2: MMM I&X Import-4	0.00100446	0.000898123	1.118	0.00088191	1.139	0.000938654	1.07
TpetraExt MueLu::A*P-2: MMM I&X RemoteMap	0.0687599	0.0558498	1.231	0.0566812	1.213	0.0583296	1.179
TpetraExt MueLu::A*P-2: MMM M5 Cmap	3.12804	3.14197	0.996	3.14992	0.993	3.11051	1.006
TpetraExt MueLu::A*P-2: MMM Newmatrix ESFC	0.0973389	0.10192	0.955	0.097512	0.998	0.0988498	0.985
TpetraExt MueLu::A*P-2: MMM Newmatrix Final Sort	0.437435	0.436569	1.002	0.436555	1.002	0.436597	1.002
TpetraExt MueLu::A*P-2: MMM Newmatrix SerialCore	2.38751	2.36614	1.009	2.36588	1.009	2.36554	1.009
TpetraExt MueLu::R*(AP)-implicit-1: MMM All I&X	0.000229359	0.000239134	0.959	0.000281811	0.814	0.000282049	0.813
TpetraExt MueLu::R*(AP)-implicit-1: MMM All Multiply	14.9407	15.0713	0.991	15.1598	0.986	15.5475	0.961
TpetraExt MueLu::R*(AP)-implicit-1: MMM All Setup	0.000519276	0.000538111	0.965	0.000603914	0.86	0.000604153	0.86
TpetraExt MueLu::R*(AP)-implicit-1: MMM I&X Alloc	0.000235319	0.000250816	0.938	0.00025034	0.94	0.00024724	0.952
TpetraExt MueLu::R*(AP)-implicit-1: MMM M5 Cmap	7.98194	8.33938	0.957	8.08079	0.988	8.19883	0.974
TpetraExt MueLu::R*(AP)-implicit-1: MMM Newmatrix ESFC	2.0716	2.47787	0.836	2.21615	0.935	2.34303	0.884
TpetraExt MueLu::R*(AP)-implicit-1: MMM Newmatrix Final Sort	0.998959	0.995499	1.003	1.03017	0.97	1.03697	0.963
TpetraExt MueLu::R*(AP)-implicit-1: MMM Newmatrix SerialCore	4.50694	4.45677	1.011	4.59923	0.98	4.59021	0.982
TpetraExt MueLu::R*(AP)-implicit-1: MMM-T AB-core	7.98331	8.3406	0.957	8.08199	0.988	8.20006	0.974
TpetraExt MueLu::R*(AP)-implicit-1: MMM-T I&X	0.00117612	0.00116396	1.01	0.00115705	1.016	0.00118971	0.989
TpetraExt MueLu::R*(AP)-implicit-1: MMM-T Transpose	1.08332	1.20838	0.897	1.18175	0.917	1.21136	0.894
TpetraExt MueLu::R*(AP)-implicit-1: MMM-T exportAndFillComplete	5.88436	5.5301	1.064	5.89653	0.998	6.13685	0.959
TpetraExt MueLu::R*(AP)-implicit-2: MMM All I&X	0.000255585	0.000287533	0.889	0.00028801	0.887	0.000283003	0.903
TpetraExt MueLu::R*(AP)-implicit-2: MMM All Multiply	3.6431	3.87573	0.94	3.82232	0.953	3.96478	0.919
TpetraExt MueLu::R*(AP)-implicit-2: MMM All Setup	0.000542641	0.000587225	0.924	0.000566721	0.958	0.00054121	1.003
TpetraExt MueLu::R*(AP)-implicit-2: MMM I&X Alloc	0.000237703	0.000254154	0.935	0.00024581	0.967	0.000258446	0.92
TpetraExt MueLu::R*(AP)-implicit-2: MMM M5 Cmap	2.20151	2.37388	0.927	2.20487	0.998	2.25229	0.977
TpetraExt MueLu::R*(AP)-implicit-2: MMM Newmatrix ESFC	1.22573	1.39525	0.879	1.22765	0.998	1.27267	0.963
TpetraExt MueLu::R*(AP)-implicit-2: MMM Newmatrix Final Sort	0.133173	0.13316	1.0	0.133385	0.998	0.133543	0.997
TpetraExt MueLu::R*(AP)-implicit-2: MMM Newmatrix SerialCore	1.12988	1.11158	1.016	1.1128	1.015	1.1135	1.015
TpetraExt MueLu::R*(AP)-implicit-2: MMM-T AB-core	2.20258	2.37489	0.927	2.20594	0.998	2.25338	0.977
TpetraExt MueLu::R*(AP)-implicit-2: MMM-T I&X	0.0015223	0.0014286	1.066	0.0014956	1.018	0.00148916	1.022
TpetraExt MueLu::R*(AP)-implicit-2: MMM-T Transpose	0.1409	0.140187	1.005	0.141362	0.997	0.141067	0.999
TpetraExt MueLu::R*(AP)-implicit-2: MMM-T exportAndFillComplete	1.34076	1.40163	0.957	1.5165	0.884	1.61158	0.832
TpetraExt MueLu::SaP-1: Jacobi All I&X	3.31384	4.05054	0.818	3.88533	0.853	3.41511	0.97
TpetraExt MueLu::SaP-1: Jacobi All Multiply	7.53955	6.67634	1.129	6.25623	1.205	6.22924	1.21
TpetraExt MueLu::SaP-1: Jacobi All Setup	0.000496149	0.000538111	0.922	0.000509739	0.973	0.000513554	0.966
TpetraExt MueLu::SaP-1: Jacobi M5 Cmap	0.110757	0.110724	1.0	0.128659	0.861	0.109782	1.009
TpetraExt MueLu::SaP-1: Jacobi Newmatrix ESFC	2.89004	1.95183	1.481	1.65228	1.749	1.6588	1.742
TpetraExt MueLu::SaP-1: Jacobi Newmatrix Final Sort	0.499618	0.479213	1.043	0.47917	1.043	0.477277	1.047
TpetraExt MueLu::SaP-1: Jacobi Newmatrix SerialCore	6.05698	4.96881	1.219	4.92681	1.229	4.93241	1.228
TpetraExt MueLu::SaP-1: MMM I&X Alloc	0.000531673	0.000603199	0.881	0.000525475	1.012	0.000528812	1.005
TpetraExt MueLu::SaP-1: MMM I&X Collective-0	0.00983739	0.0125523	0.784	0.0318065	0.309	0.00777555	1.265
TpetraExt MueLu::SaP-1: MMM I&X Import-2	0.0791609	0.0736144	1.075	0.0762317	1.038	0.0750852	1.054
TpetraExt MueLu::SaP-1: MMM I&X Import-3	1.36581	1.43132	0.954	1.83761	0.743	1.39588	0.978
TpetraExt MueLu::SaP-1: MMM I&X Import-4	0.000953436	0.000844002	1.13	0.000885248	1.077	0.000932217	1.023
TpetraExt MueLu::SaP-1: MMM I&X RemoteMap	1.88303	2.55261	0.738	1.97363	0.954	1.95574	0.963
TpetraExt MueLu::SaP-2: Jacobi All I&X	1.31656	1.32677	0.992	1.36163	0.967	1.44754	0.91
TpetraExt MueLu::SaP-2: Jacobi All Multiply	1.10379	1.1411	0.967	1.10954	0.995	1.11058	0.994
TpetraExt MueLu::SaP-2: Jacobi All Setup	0.000394583	0.000396013	0.996	0.000398636	0.99	0.000455141	0.867
TpetraExt MueLu::SaP-2: Jacobi M5 Cmap	0.0429106	0.0397716	1.079	0.0395582	1.085	0.0402765	1.065
TpetraExt MueLu::SaP-2: Jacobi Newmatrix ESFC	0.088846	0.0944414	0.941	0.0890236	0.998	0.088968	0.999
TpetraExt MueLu::SaP-2: Jacobi Newmatrix Final Sort	0.119327	0.117784	1.013	0.118089	1.01	0.118193	1.01
TpetraExt MueLu::SaP-2: Jacobi Newmatrix SerialCore	0.893115	0.909963	0.981	0.894704	0.998	0.894056	0.999
TpetraExt MueLu::SaP-2: MMM I&X Alloc	0.000495672	0.000480413	1.032	0.000483274	1.026	0.000472069	1.05
TpetraExt MueLu::SaP-2: MMM I&X Collective-0	0.00997567	0.0102584	0.972	0.00790501	1.262	0.00982666	1.015
TpetraExt MueLu::SaP-2: MMM I&X Import-2	0.0352678	0.0335939	1.05	0.0360272	0.979	0.0342643	1.029
TpetraExt MueLu::SaP-2: MMM I&X Import-3	1.11918	1.06538	1.05	1.14794	0.975	1.23169	0.909
TpetraExt MueLu::SaP-2: MMM I&X Import-4	0.000867844	0.000815153	1.065	0.000884295	0.981	0.00089407	0.971
TpetraExt MueLu::SaP-2: MMM I&X RemoteMap	0.159472	0.22084	0.722	0.175292	0.91	0.17585	0.907
create problem		0.146642		0.140803		0.139731
create solution		0.046762		0.0381038		0.0398285

jjellio commented 7 years ago

I have a job that hasn't run yet, that will evaluate MueLu construction w/out the MJ stuff enabled. This should be a better baseline than the prior data. When I get that data, I'll post again with a more fair comparison.

jjellio commented 7 years ago

Updated data. The job that profiled MueLu without the modified MJ settings is now used as a base line. This avoids some of the noise in my prior data.

I've cut out the timers matching Repartition, Zoltan2, and MultiJ. I can post the full table if you like.

A,B,C are the same experiments as above: A: value = 1 B: value = 16 C: value = 64

Min timers:

Timer Name	minT	A	speedup_A	B	speedup_B	C	speedup_C
3 - Constructing Preconditioner	117.459	113.866	1.032	107.881	1.089	110.243	1.065
MueLu: RepartitionFactory: Build	2.94445	2.90198	1.015	2.77668	1.06	2.78348	1.058
MueLu: RepartitionFactory: Build (level=1)	0.0631196	0.0790834	0.798	0.0648406	0.973	0.0650868	0.97
MueLu: RepartitionFactory: Build (level=2)	2.88767	2.82917	1.021	2.71806	1.062	2.7267	1.059
MueLu: RepartitionFactory: Build (total)	81.6485	79.8413	1.023	76.9149	1.062	77.7986	1.049
MueLu: RepartitionFactory: Build (total, level=1)	50.4218	51.114	0.986	49.7064	1.014	49.8727	1.011
MueLu: RepartitionFactory: Build (total, level=2)	31.1731	28.7232	1.085	27.1559	1.148	27.9133	1.117
MueLu: RepartitionFactory: DeterminePartitionPlacement (sub, total)	1.64123	1.67578	0.979	1.61149	1.018	1.62484	1.01
MueLu: RepartitionFactory: DeterminePartitionPlacement (sub, total, level=2)	1.63666	1.66523	0.983	1.6107	1.016	1.61715	1.012
MueLu: RepartitionFactory: Import construction (sub, total)	0.184797	0.183483	1.007	0.174675	1.058	0.172039	1.074
MueLu: RepartitionFactory: Import construction (sub, total, level=2)	0.183874	0.182521	1.007	0.173728	1.058	0.171141	1.074
MueLu: RepartitionHeuristicFactory: Build	0.146921	0.162647	0.903	0.10191	1.442	0.10067	1.459
MueLu: RepartitionHeuristicFactory: Build (level=1)	0.00120473	0.00121522	0.991	0.00119829	1.005	0.00119901	1.005
MueLu: RepartitionHeuristicFactory: Build (level=2)	0.141456	0.158279	0.894	0.0975144	1.451	0.0960703	1.472
MueLu: RepartitionHeuristicFactory: Build (total)	0.148591	0.164314	0.904	0.103575	1.435	0.102381	1.451
MueLu: RepartitionHeuristicFactory: Build (total, level=1)	0.00205803	0.00205827	1.0	0.00203872	1.009	0.00204062	1.009
MueLu: RepartitionHeuristicFactory: Build (total, level=2)	0.142606	0.159318	0.895	0.0986276	1.446	0.0971265	1.468
MueLu: Zoltan2Interface: Build	14.0188	12.4761	1.124	11.1307	1.259	11.5472	1.214
MueLu: Zoltan2Interface: Build (level=1)	0.00155997	0.00157189	0.992	0.00153112	1.019	0.00155973	1.0
MueLu: Zoltan2Interface: Build (level=2)	14.0138	12.4709	1.124	11.1258	1.26	11.5423	1.214
MueLu: Zoltan2Interface: Build (total)	14.0203	12.4776	1.124	11.1322	1.259	11.5487	1.214
MueLu: Zoltan2Interface: Build (total, level=1)	0.0023787	0.00238228	0.998	0.00233459	1.019	0.00237465	1.002
MueLu: Zoltan2Interface: Build (total, level=2)	14.0151	12.4722	1.124	11.1271	1.26	11.5436	1.214
MueLu: Zoltan2Interface: Zoltan2 multijagged (sub, total)	13.853	12.3101	1.125	10.9699	1.263	11.3904	1.216
MueLu: Zoltan2Interface: Zoltan2 multijagged (sub, total, level=2)	13.8517	12.3091	1.125	10.969	1.263	11.3892	1.216
MultiJagged - Final DistributorPlanComm		0.0		0.0		0.0
MultiJagged - Final DistributorPlanCreating		0.0		0.0		0.0
MultiJagged - Migration DistributorMigration-0		0.0		0.0		0.0
MultiJagged - Migration DistributorMigration-1		0.0		0.0		0.0
MultiJagged - Migration DistributorPlanCreating-0		0.0		0.0		0.0
MultiJagged - Migration DistributorPlanCreating-1		0.0		0.0		0.0
MultiJagged - Part_Assignment		0.0		0.0		0.0
MultiJagged - PostMigration DistributorMigration		0.00521994		0.00521016		0.00522637
MultiJagged - PostMigration DistributorPlanCreating		0.070869		0.0473049		0.0442643
MultiJagged - PreMigration DistributorMigration		0.00152302		0.00151134		0.00150895
MultiJagged - PreMigration DistributorPlanCreating		0.138623		0.102562		0.149993
MultiJagged - Problem_Free		0.0		0.0		0.0
MultiJagged - Problem_Migration-0		0.0		0.0		0.0
MultiJagged - Problem_Migration-1		0.0		0.0		0.0
MultiJagged - Problem_Partitioning		0.0		0.0		0.0
MultiJagged - Problem_Partitioning_0		0.0		0.0		0.0
MultiJagged - Problem_Partitioning_1		0.0		0.0		0.0
MultiJagged - Problem_Partitioning_2		0.0		0.0		0.0
MultiJagged - Solution_Part_Assignment		0.0		0.0		0.0
MultiJagged - Total		0.0		0.0		0.0

max timers:

Timer Name	maxT	maxT_A	maxT_speedup_A	maxT_B	maxT_speedup_B	maxT_C	maxT_speedup_C
3 - Constructing Preconditioner	117.477	113.885	1.032	107.898	1.089	110.273	1.065
MueLu: RepartitionFactory: Build	4.95159	5.0768	0.975	4.8784	1.015	4.85286	1.02
MueLu: RepartitionFactory: Build (level=1)	0.135096	0.151963	0.889	0.137552	0.982	0.135852	0.994
MueLu: RepartitionFactory: Build (level=2)	2.9034	2.846	1.02	2.73387	1.062	2.7429	1.059
MueLu: RepartitionFactory: Build (total)	96.1034	90.4532	1.062	87.3644	1.1	88.8872	1.081
MueLu: RepartitionFactory: Build (total, level=1)	50.4862	51.1803	0.986	49.7747	1.014	49.945	1.011
MueLu: RepartitionFactory: Build (total, level=2)	31.2341	28.7427	1.087	27.2206	1.147	27.9364	1.118
MueLu: RepartitionFactory: DeterminePartitionPlacement (sub, total)	2.69952	2.85444	0.946	2.72826	0.989	2.74443	0.984
MueLu: RepartitionFactory: DeterminePartitionPlacement (sub, total, level=2)	1.99914	2.00213	0.999	1.97182	1.014	1.94583	1.027
MueLu: RepartitionFactory: Import construction (sub, total)	0.451028	0.464675	0.971	0.477931	0.944	0.44793	1.007
MueLu: RepartitionFactory: Import construction (sub, total, level=2)	0.189382	0.188859	1.003	0.180045	1.052	0.17671	1.072
MueLu: RepartitionHeuristicFactory: Build	0.30403	0.308879	0.984	0.245962	1.236	0.24304	1.251
MueLu: RepartitionHeuristicFactory: Build (level=1)	0.00213647	0.00304389	0.702	0.0020771	1.029	0.00207782	1.028
MueLu: RepartitionHeuristicFactory: Build (level=2)	0.163369	0.167431	0.976	0.106676	1.531	0.104257	1.567
MueLu: RepartitionHeuristicFactory: Build (total)	0.30827	0.314034	0.982	0.251038	1.228	0.248243	1.242
MueLu: RepartitionHeuristicFactory: Build (total, level=1)	0.00348115	0.00426507	0.816	0.00330114	1.055	0.00339341	1.026
MueLu: RepartitionHeuristicFactory: Build (total, level=2)	0.164612	0.168721	0.976	0.108016	1.524	0.105598	1.559
MueLu: Zoltan2Interface: Build	18.7579	13.9386	1.346	12.5566	1.494	13.5379	1.386
MueLu: Zoltan2Interface: Build (level=1)	0.00218487	0.00267506	0.817	0.0022428	0.974	0.0022223	0.983
MueLu: Zoltan2Interface: Build (level=2)	14.0265	12.4827	1.124	11.1326	1.26	11.5518	1.214
MueLu: Zoltan2Interface: Build (total)	18.762	13.9428	1.346	12.5606	1.494	13.542	1.385
MueLu: Zoltan2Interface: Build (total, level=1)	0.0030539	0.00356054	0.858	0.00308585	0.99	0.00313807	0.973
MueLu: Zoltan2Interface: Build (total, level=2)	14.0276	12.4839	1.124	11.1337	1.26	11.5529	1.214
MueLu: Zoltan2Interface: Zoltan2 multijagged (sub, total)	18.3779	13.5781	1.353	12.1842	1.508	13.1849	1.394
MueLu: Zoltan2Interface: Zoltan2 multijagged (sub, total, level=2)	13.8618	12.3213	1.125	10.9806	1.262	11.4033	1.216
MultiJagged - Final DistributorPlanComm		0.243017		0.0908208		0.234641
MultiJagged - Final DistributorPlanCreating		0.676229		0.492985		0.764104
MultiJagged - Migration DistributorMigration-0		0.135608		0.133178		0.130844
MultiJagged - Migration DistributorMigration-1		0.151176		0.097888		0.109345
MultiJagged - Migration DistributorPlanCreating-0		1.47905		0.731299		0.836204
MultiJagged - Migration DistributorPlanCreating-1		0.172276		0.0601864		0.0962286
MultiJagged - Part_Assignment		0.772875		0.555005		0.866364
MultiJagged - PostMigration DistributorMigration		0.0456626		0.0574806		0.0547552
MultiJagged - PostMigration DistributorPlanCreating		12.6652		11.3486		12.2846
MultiJagged - PreMigration DistributorMigration		0.0691514		0.076077		0.0754247
MultiJagged - PreMigration DistributorPlanCreating		0.687274		0.463595		0.533804
MultiJagged - Problem_Free		0.000704765		0.000634909		0.000650406
MultiJagged - Problem_Migration-0		9.75678		9.27724		9.77918
MultiJagged - Problem_Migration-1		0.704301		0.620316		0.641442
MultiJagged - Problem_Partitioning		11.5054		10.5979		11.1695
MultiJagged - Problem_Partitioning_0		9.98574		9.548		10.0623
MultiJagged - Problem_Partitioning_1		1.27529		0.875352		0.924565
MultiJagged - Problem_Partitioning_2		0.239334		0.124609		0.144092
MultiJagged - Solution_Part_Assignment		0.000112772		9.87053e-05		0.00011611
MultiJagged - Total		12.2412		10.9856		11.921

aprokop commented 7 years ago

@jjellio tl;dr?

jjellio commented 7 years ago

Not sure really. This was with a Laplace3D problem (7nnz per row and 1968^3 rows) pwxy used Brick3D which is 27 nnz per row. The problem I ran is much larger, but not as dense.

The gains from Mehmet's work show up in the 'MueLu: Zoltan2*' timers, but they are not as large as @pwxy.

MJ value =1 seems inferior to 16 or 64 MJ value =1 seems to slow down the code in some regions compared to not enabling any MJ modifications at all

The gains from value = 16 or 64 appear to neutralize or improve the code where value=1 came up with a speedup < 1.

Overall, the MJ work (value =1 or 16/64) always nets an improvement. The parts that speedup outweigh the parts that slowed down. @mndevec has the full verbose output logs and is looking at how sharp the drop in processors is.

What complicates comparing with prior runs is that I've been able to reduce system noise significantly resulting in overall speedups of nearly 4x.

It would be nice to understand what is different between pwxy and my own runs. I suspect he is performing more smaller setups, where I perform 20 huge monolithic setups. (or that is what I observed when I looked at Drekar). The problem sizes I use target having around 200k unknowns per core.

mndevec commented 7 years ago

@jjellio I made another push to MJ. premigration option is now binary, any positive number is equivalent to 1. Could you give it another run? (Just with mj_pregmiration_option = 1)

In your runs, multigrid reduces the number of ranks from 32K to 24K in the first repartitioning. MJ premigration reduces the number of ranks from 32K to 24K, therefore it does not really make that much difference. In Paul's runs this reduction is from 132K to 1K ranks, the improvements become much more visible in that case. This reduction is also visible in your runs at the second repartitoning where the number of ranks is reduced much more.

With the latest push, I added a new parameter, mj_premigration_coordinate_count that defaults to 32000 (you do not need to set that one for now). Premigration will be done in a way that each rank will have about 32000 coordinates during MJ runs. This should reduce the used number of ranks from 24000 to 750, hopefully it should help with the scaling.

jjellio commented 7 years ago

Yes, I'll pull and update my builds.

It is important to note: I've been fiddling with system parameters, and atleast for the case of Tpetra's SpMV I am now observing the best single node performance from a single process on a KNL node. (e.g., ~256 threads). This is good news, and means I've address several of the problems I ran into on the first Cori runs. The stuff I've twiddled with substantially improves onnode memory bandwidth and cache behavior. I don't think it will impact what you've done, but I do expect to see very different performance on my next batch of runs than I observed on my March/May Cori datasets (not too different from the more recent runs I posted here though, those used a subset of the parameters I've been testing).

aprokop commented 7 years ago

The stuff I've twiddled with substantially improves onnode memory bandwidth and cache behavior.

@jjellio How do you plan to disseminate your experience?

jjellio commented 7 years ago

Should be a slide deck. Right now, I am working on the scripts that score and aggregate the various options. I need to know which set of parameters to choose as the baseline before I can redo the runs.

aprokop commented 7 years ago

@jjellio @mndevec Any updates on Zoltan2 Cori runs?

jjellio commented 7 years ago

Not yet. Baseline Muelu data (w/out MJ work) is nearly finished.

An MJ run should go into the que today.

aprokop commented 6 years ago

Has the team come to any resolution on this issue?

kddevin commented 6 years ago

No recent news. Changes made by @mndevec helped @pwxy 's test cases. @jjellio , did you see anything interesting?

jjellio commented 6 years ago

We gathered some data, but for large Laplace3D problem, we did not see any gains from Mehmet's work. We talked and came to the conclusion that this is probably a problem specific tuning parameter.

pwxy commented 6 years ago

As mentioned back in August 2017, "mj_premigration_option"=1 provides a very significant improvement in Z2 MJ time for Drekar, especially at very large scales. I forgot to add these results from September 2017.

2.5 billion row Drekar Poisson problem run from 131,072 to 1,048,576 MPI processes on BG/Q (16 cores per compute node, 1 MPI per core). 27-point stencil, 27 nnz/row. Times below in seconds.

MPI	premig=0	premig=1	reduction
131K	2.1	0.15	14x
262K		0.25
524K	12.3	0.46	27x
1M	26.7	0.84	32x

"mj_premigration_option"=1 reduced MJ time by 32x for 1M MPI processes

@mndevec thanks for the huge reduction in MJ time!

mndevec commented 6 years ago

@jjellio

How are the timings? Was there a slow-down or were the times unaffected? How many processors were you running it on, was it the same scale as Paul's experiments?

github-actions[bot] commented 3 years ago

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity. If you would like to keep this issue open please add a comment and remove the MARKED_FOR_CLOSURE label. If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.

github-actions[bot] commented 3 years ago

This issue was closed due to inactivity for 395 days.

trilinos / Trilinos

Zoltan2: Possible scaling issue with MueLu/Z2multijagged for MueLu coarse level repartitioning? #1538