NVIDIA / AMGX

Distributed multigrid linear solver library on GPU
482 stars 139 forks source link

Seeking advice for setting up solver for finite element #303

Open kiendangtor opened 5 months ago

kiendangtor commented 5 months ago

Environment information:

OS: Windows 11 CUDA runtime: CUDA 12.4. AMGX version or commit hash: 2.5.0 NVIDIA driver: 551.78 NVIDIA GPU: NVIDIA RTX 3060 Ti

AMGX solver configuration: This is one of AMG configuration that I tried to use but failed to converged. { "config_version": 2, "solver": { "matrix_coloring_scheme": "MIN_MAX", "max_uncolored_percentage": 0.15, "algorithm": "AGGREGATION", "obtain_timings": 1, "solver": "AMG", "smoother": "MULTICOLOR_DILU", "print_solve_stats": 1, "presweeps": 1, "selector": "SIZE_2", "coarsest_sweeps": 2, "max_iters": 1000, "monitor_residual": 1, "scope": "main", "max_levels": 1000, "postsweeps": 1, "tolerance": 0.1, "print_grid_stats": 1, "norm": "L1", "cycle": "V" } }

Matrix is attached

Reproduction steps

I ran this with the command line as in the quick start document :"amgx_capi -m identity_system.mtx -c AGGREGATION_DILU.json

It got diverged. Samte thing with AMG_CLASSICAL_CG.json. Using AMG_AGGRREGATION_CG it manage to converge but very slow

If I use PCG_AGGREGATION_JACOBI i got the best convergence speed.

Can you please tell me which option should I choose for the type of matrix that I attached? This is basically a 10 nodes tet mesh generated from a 3D solid dishes.

Thanks,

Kien

coarse.zip

coarse.zip

marsaev commented 3 months ago

@kiendangtor I apologize for the slow response.

This matrix is quite small to exploit full GPU power. Would you consider direct solve approach here?

Other than that, configurations from provided examples are typical for this type of meshes (of course also depending on what are you trying to solve). I had in mind something like "config scanner" to find best solver options for given matrix, but no ETA on that.

kiendangtor commented 2 months ago

Thank you for your reply. "config scanner" could be awesome. I had bigger mesh with 20mil of non zero but to make the diagnosis easier I attached a very coarse mesh. But the behavior are the same. I was able to successfully run the solver with many configs but all the one that has "MULTICOLOR_DILU" did not work either as a smoother or as preconditioner. Do we need to provide graph separately? I look at the code and it doesnt seem like it need that.

marsaev commented 2 months ago

Understood, thanks. Right, starting with smaller matrix is often easier to put aside what's clearly not working. Just wanted to clarify some things 1) Do we need to provide graph separately? At this moment the linear system (A, rhs, optional x_0) is the only input AMGX is working with. 2) I was able to successfully run the solver with many configs So for the larger system (i.e. 20m NNZ, as you have mentioned above) many configs work, but they are still not fast enough? Or what kind of problem you experience there? Can you provide output of the best performing config for your larger case? 3) all the one that has "MULTICOLOR_DILU" did not work either as a smoother or as preconditioner. Can you elaborate what you mean here? In general, there is no reason to always use stronger smoother (like ILU), since they can be significantly slower than other smoothers, and we can try to compensate convergence rate by adjusting multigrid structure.

Thanks,

kiendangtor commented 2 months ago

Thank you for your fast response. I have upload the larger system here: https://we.tl/t-ymNm1kBuan

  1. My issue is that the CG solver without preconditioner is the best option so far in my case.. Here is my best configuration (CG without Precond, took ~ 6s) { "config_version": 2, "solver": { "solver": "CG", "print_solve_stats": 1, "obtain_timings": 1, "max_iters": 2000, "monitor_residual": 1, "convergence": "RELATIVE_INI_CORE",
    "scope": "main", "tolerance": 1e-6, "norm": "L2" } }

Here is the output: AMGX version 2.5.0 Built on Apr 25 2024, 13:20:11 Compiled with CUDA Runtime 12.4, using CUDA driver 12.4 iter Mem Usage (GB) residual rate

        Ini                   0   1.353886e+01
          0                   0   1.262998e+01         0.9329
          1              0.0000   1.344963e+01         1.0649
          2              0.0000   1.341992e+01         0.9978
          3              0.0000   1.287418e+01         0.9593
          4              0.0000   1.291408e+01         1.0031
          5              0.0000   1.116468e+01         0.8645
          6              0.0000   1.006132e+01         0.9012
          7              0.0000   9.782631e+00         0.9723
          8              0.0000   9.901049e+00         1.0121
          9              0.0000   9.289660e+00         0.9383
         10              0.0000   8.494891e+00         0.9144
         11              0.0000   8.746236e+00         1.0296
         12              0.0000   9.184710e+00         1.0501
         13              0.0000   8.967623e+00         0.9764
         14              0.0000   8.210301e+00         0.9155
         15              0.0000   7.656633e+00         0.9326
         16              0.0000   7.515435e+00         0.9816
         17              0.0000   7.752882e+00         1.0316
         18              0.0000   7.573552e+00         0.9769
         19              0.0000   6.960823e+00         0.9191
         20              0.0000   6.493094e+00         0.9328
         21              0.0000   6.856289e+00         1.0559
         22              0.0000   7.807127e+00         1.1387
         23              0.0000   7.382842e+00         0.9457
         24              0.0000   6.450630e+00         0.8737
         25              0.0000   6.198494e+00         0.9609
         26              0.0000   6.606438e+00         1.0658
         27              0.0000   6.915229e+00         1.0467
         28              0.0000   6.720466e+00         0.9718
         29              0.0000   6.636989e+00         0.9876
         30              0.0000   6.287094e+00         0.9473
         31              0.0000   5.595264e+00         0.8900
         32              0.0000   5.520952e+00         0.9867
         33              0.0000   5.446880e+00         0.9866
         34              0.0000   5.337425e+00         0.9799
         35              0.0000   5.085706e+00         0.9528
         36              0.0000   4.873419e+00         0.9583
         37              0.0000   4.674906e+00         0.9593
         38              0.0000   4.506826e+00         0.9640
         39              0.0000   4.389167e+00         0.9739
         40              0.0000   4.533550e+00         1.0329
         41              0.0000   4.536435e+00         1.0006
         42              0.0000   4.155694e+00         0.9161
         43              0.0000   4.163612e+00         1.0019
         44              0.0000   4.321235e+00         1.0379
         45              0.0000   3.934455e+00         0.9105
         46              0.0000   3.598542e+00         0.9146
         47              0.0000   3.606063e+00         1.0021
         48              0.0000   3.828375e+00         1.0616
         49              0.0000   3.652129e+00         0.9540
         50              0.0000   3.340484e+00         0.9147
         51              0.0000   3.341184e+00         1.0002
         52              0.0000   3.360057e+00         1.0056
         53              0.0000   3.161396e+00         0.9409
         54              0.0000   3.171314e+00         1.0031
         55              0.0000   3.069862e+00         0.9680
         56              0.0000   2.932093e+00         0.9551
         57              0.0000   2.925210e+00         0.9977
         58              0.0000   2.768549e+00         0.9464
         59              0.0000   2.684689e+00         0.9697
         60              0.0000   2.733094e+00         1.0180
         61              0.0000   2.531500e+00         0.9262
         62              0.0000   2.409799e+00         0.9519
         63              0.0000   2.575164e+00         1.0686
         64              0.0000   2.486035e+00         0.9654
         65              0.0000   2.201733e+00         0.8856
         66              0.0000   2.209115e+00         1.0034
         67              0.0000   2.206543e+00         0.9988
         68              0.0000   2.200250e+00         0.9971
         69              0.0000   2.254593e+00         1.0247
         70              0.0000   2.201090e+00         0.9763
         71              0.0000   2.113318e+00         0.9601
         72              0.0000   2.080655e+00         0.9845
         73              0.0000   2.179618e+00         1.0476
         74              0.0000   2.060706e+00         0.9454
         75              0.0000   2.057177e+00         0.9983
         76              0.0000   2.110698e+00         1.0260
         77              0.0000   1.945716e+00         0.9218
         78              0.0000   1.900318e+00         0.9767
         79              0.0000   1.922642e+00         1.0117
         80              0.0000   1.836647e+00         0.9553
         81              0.0000   1.834999e+00         0.9991
         82              0.0000   1.910327e+00         1.0411
         83              0.0000   1.781650e+00         0.9326
         84              0.0000   1.714393e+00         0.9622
         85              0.0000   1.695226e+00         0.9888
         86              0.0000   1.698762e+00         1.0021
         87              0.0000   1.756629e+00         1.0341
         88              0.0000   1.720690e+00         0.9795
         89              0.0000   1.655724e+00         0.9622
         90              0.0000   1.667694e+00         1.0072
         91              0.0000   1.695344e+00         1.0166
         92              0.0000   1.588701e+00         0.9371
         93              0.0000   1.541870e+00         0.9705
         94              0.0000   1.582440e+00         1.0263
         95              0.0000   1.581971e+00         0.9997
         96              0.0000   1.522868e+00         0.9626
         97              0.0000   1.524990e+00         1.0014
         98              0.0000   1.475319e+00         0.9674
         99              0.0000   1.471768e+00         0.9976
        100              0.0000   1.501872e+00         1.0205
        101              0.0000   1.425984e+00         0.9495
        102              0.0000   1.397138e+00         0.9798
        103              0.0000   1.411637e+00         1.0104
        104              0.0000   1.376674e+00         0.9752
        105              0.0000   1.334801e+00         0.9696
        106              0.0000   1.331577e+00         0.9976
        107              0.0000   1.352925e+00         1.0160
        108              0.0000   1.282638e+00         0.9480
        109              0.0000   1.289760e+00         1.0056
        110              0.0000   1.311367e+00         1.0168
        111              0.0000   1.271219e+00         0.9694
        112              0.0000   1.256974e+00         0.9888
        113              0.0000   1.279412e+00         1.0179
        114              0.0000   1.257423e+00         0.9828
        115              0.0000   1.231465e+00         0.9794
        116              0.0000   1.235452e+00         1.0032
        117              0.0000   1.186845e+00         0.9607
        118              0.0000   1.158272e+00         0.9759
        119              0.0000   1.163486e+00         1.0045
        120              0.0000   1.146347e+00         0.9853
        121              0.0000   1.128234e+00         0.9842
        122              0.0000   1.151797e+00         1.0209
        123              0.0000   1.163217e+00         1.0099
        124              0.0000   1.141058e+00         0.9810
        125              0.0000   1.130028e+00         0.9903
        126              0.0000   1.090592e+00         0.9651
        127              0.0000   1.084568e+00         0.9945
        128              0.0000   1.052836e+00         0.9707
        129              0.0000   1.031181e+00         0.9794
        130              0.0000   1.057783e+00         1.0258
        131              0.0000   1.037753e+00         0.9811
        132              0.0000   1.003208e+00         0.9667
        133              0.0000   1.009502e+00         1.0063
        134              0.0000   1.015455e+00         1.0059
        135              0.0000   9.903361e-01         0.9753
        136              0.0000   9.782567e-01         0.9878
        137              0.0000   9.792711e-01         1.0010
        138              0.0000   9.522720e-01         0.9724
        139              0.0000   9.373981e-01         0.9844
        140              0.0000   9.393499e-01         1.0021
        141              0.0000   9.349063e-01         0.9953
        142              0.0000   9.307010e-01         0.9955
        143              0.0000   9.104812e-01         0.9783
        144              0.0000   9.000397e-01         0.9885
        145              0.0000   8.931340e-01         0.9923
        146              0.0000   8.813223e-01         0.9868
        147              0.0000   8.854138e-01         1.0046
        148              0.0000   8.714043e-01         0.9842
        149              0.0000   8.596665e-01         0.9865
        150              0.0000   8.388330e-01         0.9758
        151              0.0000   8.474289e-01         1.0102
        152              0.0000   8.478095e-01         1.0004
        153              0.0000   8.403001e-01         0.9911
        154              0.0000   8.205706e-01         0.9765
        155              0.0000   8.303758e-01         1.0119
        156              0.0000   8.393402e-01         1.0108
        157              0.0000   8.239402e-01         0.9817
        158              0.0000   8.310023e-01         1.0086
        159              0.0000   8.476804e-01         1.0201
        160              0.0000   8.022486e-01         0.9464
        161              0.0000   7.757737e-01         0.9670
        162              0.0000   7.889112e-01         1.0169
        163              0.0000   7.860369e-01         0.9964
        164              0.0000   7.505643e-01         0.9549
        165              0.0000   7.480393e-01         0.9966
        166              0.0000   7.573304e-01         1.0124
        167              0.0000   7.290384e-01         0.9626
        168              0.0000   7.477997e-01         1.0257
        169              0.0000   7.526946e-01         1.0065
        170              0.0000   7.233232e-01         0.9610
        171              0.0000   7.462722e-01         1.0317
        172              0.0000   7.483545e-01         1.0028
        173              0.0000   7.120085e-01         0.9514
        174              0.0000   7.181118e-01         1.0086
        175              0.0000   7.167547e-01         0.9981
        176              0.0000   7.207807e-01         1.0056
        177              0.0000   7.161876e-01         0.9936
        178              0.0000   6.931766e-01         0.9679
        179              0.0000   6.684345e-01         0.9643
        180              0.0000   6.566061e-01         0.9823
        181              0.0000   6.584297e-01         1.0028
        182              0.0000   6.757532e-01         1.0263
        183              0.0000   6.572882e-01         0.9727
        184              0.0000   6.698679e-01         1.0191
        185              0.0000   6.785304e-01         1.0129
        186              0.0000   6.390069e-01         0.9418
        187              0.0000   6.447940e-01         1.0091
        188              0.0000   6.370597e-01         0.9880
        189              0.0000   6.382035e-01         1.0018
        190              0.0000   6.326952e-01         0.9914
        191              0.0000   6.173935e-01         0.9758
        192              0.0000   6.126585e-01         0.9923
        193              0.0000   6.200593e-01         1.0121
        194              0.0000   6.183008e-01         0.9972
        195              0.0000   6.195359e-01         1.0020
        196              0.0000   6.215696e-01         1.0033
        197              0.0000   6.009018e-01         0.9667
        198              0.0000   6.136047e-01         1.0211
        199              0.0000   6.118257e-01         0.9971
        200              0.0000   6.168415e-01         1.0082
        201              0.0000   6.159987e-01         0.9986
        202              0.0000   5.853574e-01         0.9503
        203              0.0000   5.884180e-01         1.0052
        204              0.0000   6.030999e-01         1.0250
        205              0.0000   5.890910e-01         0.9768
        206              0.0000   5.823547e-01         0.9886
        207              0.0000   5.855304e-01         1.0055
        208              0.0000   5.724553e-01         0.9777
        209              0.0000   5.614496e-01         0.9808
        210              0.0000   5.762443e-01         1.0264
        211              0.0000   5.673486e-01         0.9846
        212              0.0000   5.650837e-01         0.9960
        213              0.0000   5.709320e-01         1.0103
        214              0.0000   5.884823e-01         1.0307
        215              0.0000   5.737960e-01         0.9750
        216              0.0000   5.553936e-01         0.9679
        217              0.0000   5.511891e-01         0.9924
        218              0.0000   5.436915e-01         0.9864
        219              0.0000   5.523357e-01         1.0159
        220              0.0000   5.481563e-01         0.9924
        221              0.0000   5.295897e-01         0.9661
        222              0.0000   5.276222e-01         0.9963
        223              0.0000   5.382623e-01         1.0202
        224              0.0000   5.356678e-01         0.9952
        225              0.0000   5.328757e-01         0.9948
        226              0.0000   5.263735e-01         0.9878
        227              0.0000   5.179160e-01         0.9839
        228              0.0000   5.029985e-01         0.9712
        229              0.0000   4.939617e-01         0.9820
        230              0.0000   4.967452e-01         1.0056
        231              0.0000   4.767040e-01         0.9597
        232              0.0000   4.719018e-01         0.9899
        233              0.0000   4.827788e-01         1.0230
        234              0.0000   4.641570e-01         0.9614
        235              0.0000   4.580175e-01         0.9868
        236              0.0000   4.626222e-01         1.0101
        237              0.0000   4.577362e-01         0.9894
        238              0.0000   4.635887e-01         1.0128
        239              0.0000   4.624222e-01         0.9975
        240              0.0000   4.438046e-01         0.9597
        241              0.0000   4.463173e-01         1.0057
        242              0.0000   4.554095e-01         1.0204
        243              0.0000   4.301062e-01         0.9444
        244              0.0000   4.336583e-01         1.0083
        245              0.0000   4.284497e-01         0.9880
        246              0.0000   4.189278e-01         0.9778
        247              0.0000   4.373496e-01         1.0440
        248              0.0000   4.332623e-01         0.9907
        249              0.0000   4.345460e-01         1.0030
        250              0.0000   4.393239e-01         1.0110
        251              0.0000   4.207781e-01         0.9578
        252              0.0000   4.332030e-01         1.0295
        253              0.0000   4.239820e-01         0.9787
        254              0.0000   4.211489e-01         0.9933
        255              0.0000   4.192285e-01         0.9954
        256              0.0000   4.034146e-01         0.9623
        257              0.0000   4.089019e-01         1.0136
        258              0.0000   4.069869e-01         0.9953
        259              0.0000   4.020965e-01         0.9880
        260              0.0000   4.101955e-01         1.0201
        261              0.0000   4.054649e-01         0.9885
        262              0.0000   3.963023e-01         0.9774
        263              0.0000   3.990551e-01         1.0069
        264              0.0000   3.803718e-01         0.9532
        265              0.0000   3.792435e-01         0.9970
        266              0.0000   3.828624e-01         1.0095
        267              0.0000   3.575243e-01         0.9338
        268              0.0000   3.442248e-01         0.9628
        269              0.0000   3.424394e-01         0.9948
        270              0.0000   3.271007e-01         0.9552
        271              0.0000   3.143270e-01         0.9609
        272              0.0000   3.144425e-01         1.0004
        273              0.0000   3.037625e-01         0.9660
        274              0.0000   2.861743e-01         0.9421
        275              0.0000   2.769139e-01         0.9676
        276              0.0000   2.671295e-01         0.9647
        277              0.0000   2.532142e-01         0.9479
        278              0.0000   2.375813e-01         0.9383
        279              0.0000   2.234230e-01         0.9404
        280              0.0000   2.069208e-01         0.9261
        281              0.0000   1.929073e-01         0.9323
        282              0.0000   1.848349e-01         0.9582
        283              0.0000   1.809836e-01         0.9792
        284              0.0000   1.736374e-01         0.9594
        285              0.0000   1.527659e-01         0.8798
        286              0.0000   1.417858e-01         0.9281
        287              0.0000   1.315394e-01         0.9277
        288              0.0000   1.235922e-01         0.9396
        289              0.0000   1.162488e-01         0.9406
        290              0.0000   1.119325e-01         0.9629
        291              0.0000   1.068759e-01         0.9548
        292              0.0000   9.955425e-02         0.9315
        293              0.0000   9.073929e-02         0.9115
        294              0.0000   8.643774e-02         0.9526
        295              0.0000   8.299320e-02         0.9602
        296              0.0000   8.080298e-02         0.9736
        297              0.0000   7.763006e-02         0.9607
        298              0.0000   7.273561e-02         0.9370
        299              0.0000   7.232574e-02         0.9944
        300              0.0000   6.658044e-02         0.9206
        301              0.0000   6.364191e-02         0.9559
        302              0.0000   6.233454e-02         0.9795
        303              0.0000   6.132779e-02         0.9838
        304              0.0000   5.801469e-02         0.9460
        305              0.0000   5.795389e-02         0.9990
        306              0.0000   5.633932e-02         0.9721
        307              0.0000   5.434833e-02         0.9647
        308              0.0000   5.382773e-02         0.9904
        309              0.0000   5.311001e-02         0.9867
        310              0.0000   5.027714e-02         0.9467
        311              0.0000   4.726738e-02         0.9401
        312              0.0000   4.629796e-02         0.9795
        313              0.0000   4.521235e-02         0.9766
        314              0.0000   4.273949e-02         0.9453
        315              0.0000   4.155467e-02         0.9723
        316              0.0000   4.217253e-02         1.0149
        317              0.0000   3.988379e-02         0.9457
        318              0.0000   3.859674e-02         0.9677
        319              0.0000   3.897464e-02         1.0098
        320              0.0000   3.746127e-02         0.9612
        321              0.0000   3.647192e-02         0.9736
        322              0.0000   3.558145e-02         0.9756
        323              0.0000   3.408042e-02         0.9578
        324              0.0000   3.336385e-02         0.9790
        325              0.0000   3.280365e-02         0.9832
        326              0.0000   3.194300e-02         0.9738
        327              0.0000   3.162545e-02         0.9901
        328              0.0000   3.207835e-02         1.0143
        329              0.0000   3.056155e-02         0.9527
        330              0.0000   2.968799e-02         0.9714
        331              0.0000   2.903696e-02         0.9781
        332              0.0000   2.764549e-02         0.9521
        333              0.0000   2.757215e-02         0.9973
        334              0.0000   2.787487e-02         1.0110
        335              0.0000   2.687063e-02         0.9640
        336              0.0000   2.648189e-02         0.9855
        337              0.0000   2.687394e-02         1.0148
        338              0.0000   2.693819e-02         1.0024
        339              0.0000   2.627692e-02         0.9755
        340              0.0000   2.556245e-02         0.9728
        341              0.0000   2.536271e-02         0.9922
        342              0.0000   2.513220e-02         0.9909
        343              0.0000   2.541615e-02         1.0113
        344              0.0000   2.598969e-02         1.0226
        345              0.0000   2.565233e-02         0.9870
        346              0.0000   2.455921e-02         0.9574
        347              0.0000   2.338488e-02         0.9522
        348              0.0000   2.328266e-02         0.9956
        349              0.0000   2.367848e-02         1.0170
        350              0.0000   2.278399e-02         0.9622
        351              0.0000   2.238886e-02         0.9827
        352              0.0000   2.245145e-02         1.0028
        353              0.0000   2.184556e-02         0.9730
        354              0.0000   2.186281e-02         1.0008
        355              0.0000   2.197417e-02         1.0051
        356              0.0000   2.193714e-02         0.9983
        357              0.0000   2.215427e-02         1.0099
        358              0.0000   2.194692e-02         0.9906
        359              0.0000   2.112797e-02         0.9627
        360              0.0000   2.072562e-02         0.9810
        361              0.0000   2.087301e-02         1.0071
        362              0.0000   2.067195e-02         0.9904
        363              0.0000   2.124003e-02         1.0275
        364              0.0000   2.074068e-02         0.9765
        365              0.0000   1.932746e-02         0.9319
        366              0.0000   1.964174e-02         1.0163
        367              0.0000   1.952642e-02         0.9941
        368              0.0000   1.912590e-02         0.9795
        369              0.0000   1.956800e-02         1.0231
        370              0.0000   1.892145e-02         0.9670
        371              0.0000   1.880505e-02         0.9938
        372              0.0000   1.935910e-02         1.0295
        373              0.0000   1.886556e-02         0.9745
        374              0.0000   1.892023e-02         1.0029
        375              0.0000   1.862439e-02         0.9844
        376              0.0000   1.825186e-02         0.9800
        377              0.0000   1.813391e-02         0.9935
        378              0.0000   1.811987e-02         0.9992
        379              0.0000   1.783458e-02         0.9843
        380              0.0000   1.785637e-02         1.0012
        381              0.0000   1.832706e-02         1.0264
        382              0.0000   1.819248e-02         0.9927
        383              0.0000   1.717503e-02         0.9441
        384              0.0000   1.694985e-02         0.9869
        385              0.0000   1.674220e-02         0.9877
        386              0.0000   1.661223e-02         0.9922
        387              0.0000   1.686434e-02         1.0152
        388              0.0000   1.687965e-02         1.0009
        389              0.0000   1.698021e-02         1.0060
        390              0.0000   1.645316e-02         0.9690
        391              0.0000   1.632727e-02         0.9923
        392              0.0000   1.677217e-02         1.0272
        393              0.0000   1.567514e-02         0.9346
        394              0.0000   1.549359e-02         0.9884
        395              0.0000   1.568206e-02         1.0122
        396              0.0000   1.544631e-02         0.9850
        397              0.0000   1.578358e-02         1.0218
        398              0.0000   1.582254e-02         1.0025
        399              0.0000   1.573691e-02         0.9946
        400              0.0000   1.550577e-02         0.9853
        401              0.0000   1.548083e-02         0.9984
        402              0.0000   1.535429e-02         0.9918
        403              0.0000   1.572514e-02         1.0242
        404              0.0000   1.604480e-02         1.0203
        405              0.0000   1.601420e-02         0.9981
        406              0.0000   1.585728e-02         0.9902
        407              0.0000   1.531246e-02         0.9656
        408              0.0000   1.513600e-02         0.9885
        409              0.0000   1.563596e-02         1.0330
        410              0.0000   1.568023e-02         1.0028
        411              0.0000   1.608473e-02         1.0258
        412              0.0000   1.529724e-02         0.9510
        413              0.0000   1.491083e-02         0.9747
        414              0.0000   1.598739e-02         1.0722
        415              0.0000   1.578798e-02         0.9875
        416              0.0000   1.515374e-02         0.9598
        417              0.0000   1.520244e-02         1.0032
        418              0.0000   1.470155e-02         0.9671
        419              0.0000   1.456586e-02         0.9908
        420              0.0000   1.469726e-02         1.0090
        421              0.0000   1.468928e-02         0.9995
        422              0.0000   1.437507e-02         0.9786
        423              0.0000   1.442039e-02         1.0032
        424              0.0000   1.458873e-02         1.0117
        425              0.0000   1.410793e-02         0.9670
        426              0.0000   1.374792e-02         0.9745
        427              0.0000   1.376393e-02         1.0012
        428              0.0000   1.362299e-02         0.9898
        429              0.0000   1.357906e-02         0.9968
        430              0.0000   1.374911e-02         1.0125
        431              0.0000   1.360199e-02         0.9893
        432              0.0000   1.310474e-02         0.9634
        433              0.0000   1.307918e-02         0.9980
        434              0.0000   1.313347e-02         1.0042
        435              0.0000   1.338300e-02         1.0190
        436              0.0000   1.370956e-02         1.0244
        437              0.0000   1.287422e-02         0.9391
        438              0.0000   1.235340e-02         0.9595
        439              0.0000   1.245901e-02         1.0085
        440              0.0000   1.242344e-02         0.9971
        441              0.0000   1.247323e-02         1.0040
        442              0.0000   1.298080e-02         1.0407
        443              0.0000   1.291163e-02         0.9947
        444              0.0000   1.274599e-02         0.9872
        445              0.0000   1.255987e-02         0.9854
        446              0.0000   1.237453e-02         0.9852
        447              0.0000   1.247371e-02         1.0080
        448              0.0000   1.263250e-02         1.0127
        449              0.0000   1.271840e-02         1.0068
        450              0.0000   1.222111e-02         0.9609
        451              0.0000   1.199371e-02         0.9814
        452              0.0000   1.223683e-02         1.0203
        453              0.0000   1.206892e-02         0.9863
        454              0.0000   1.180749e-02         0.9783
        455              0.0000   1.122894e-02         0.9510
        456              0.0000   1.112267e-02         0.9905
        457              0.0000   1.132348e-02         1.0181
        458              0.0000   1.109869e-02         0.9801
        459              0.0000   1.095065e-02         0.9867
        460              0.0000   1.078360e-02         0.9847
        461              0.0000   1.050138e-02         0.9738
        462              0.0000   1.037224e-02         0.9877
        463              0.0000   1.042789e-02         1.0054
        464              0.0000   1.052245e-02         1.0091
        465              0.0000   1.026729e-02         0.9758
        466              0.0000   9.842529e-03         0.9586
        467              0.0000   9.732925e-03         0.9889
        468              0.0000   9.525361e-03         0.9787
        469              0.0000   9.552203e-03         1.0028
        470              0.0000   9.502360e-03         0.9948
        471              0.0000   9.599928e-03         1.0103
        472              0.0000   9.534268e-03         0.9932
        473              0.0000   9.076386e-03         0.9520
        474              0.0000   8.959994e-03         0.9872
        475              0.0000   8.939976e-03         0.9978
        476              0.0000   8.795774e-03         0.9839
        477              0.0000   8.727889e-03         0.9923
        478              0.0000   8.782370e-03         1.0062
        479              0.0000   8.983462e-03         1.0229
        480              0.0000   9.075476e-03         1.0102
        481              0.0000   8.419634e-03         0.9277
        482              0.0000   8.401494e-03         0.9978
        483              0.0000   8.477547e-03         1.0091
        484              0.0000   8.349500e-03         0.9849
        485              0.0000   8.223551e-03         0.9849
        486              0.0000   8.072738e-03         0.9817
        487              0.0000   7.854028e-03         0.9729
        488              0.0000   7.722498e-03         0.9833
        489              0.0000   7.667248e-03         0.9928
        490              0.0000   7.634887e-03         0.9958
        491              0.0000   7.656882e-03         1.0029
        492              0.0000   7.697057e-03         1.0052
        493              0.0000   7.669751e-03         0.9965
        494              0.0000   7.357037e-03         0.9592
        495              0.0000   7.097171e-03         0.9647
        496              0.0000   7.206420e-03         1.0154
        497              0.0000   7.076045e-03         0.9819
        498              0.0000   6.735162e-03         0.9518
        499              0.0000   6.765319e-03         1.0045
        500              0.0000   6.794066e-03         1.0042
        501              0.0000   6.688231e-03         0.9844
        502              0.0000   6.849964e-03         1.0242
        503              0.0000   6.645858e-03         0.9702
        504              0.0000   6.440888e-03         0.9692
        505              0.0000   6.320286e-03         0.9813
        506              0.0000   6.248014e-03         0.9886
        507              0.0000   6.337786e-03         1.0144
        508              0.0000   6.349945e-03         1.0019
        509              0.0000   6.349234e-03         0.9999
        510              0.0000   6.388300e-03         1.0062
        511              0.0000   6.395770e-03         1.0012
        512              0.0000   6.289377e-03         0.9834
        513              0.0000   6.241988e-03         0.9925
        514              0.0000   6.170202e-03         0.9885
        515              0.0000   6.312940e-03         1.0231
        516              0.0000   6.636119e-03         1.0512
        517              0.0000   6.341295e-03         0.9556
        518              0.0000   6.353455e-03         1.0019
        519              0.0000   6.296889e-03         0.9911
        520              0.0000   6.241926e-03         0.9913
        521              0.0000   6.386142e-03         1.0231
        522              0.0000   6.247452e-03         0.9783
        523              0.0000   6.186063e-03         0.9902
        524              0.0000   6.187890e-03         1.0003
        525              0.0000   6.105420e-03         0.9867
        526              0.0000   6.226842e-03         1.0199
        527              0.0000   6.319307e-03         1.0148
        528              0.0000   6.334320e-03         1.0024
        529              0.0000   6.464382e-03         1.0205
        530              0.0000   6.301481e-03         0.9748
        531              0.0000   6.273641e-03         0.9956
        532              0.0000   6.388512e-03         1.0183
        533              0.0000   6.345122e-03         0.9932
        534              0.0000   6.208837e-03         0.9785
        535              0.0000   6.146809e-03         0.9900
        536              0.0000   6.202854e-03         1.0091
        537              0.0000   6.234117e-03         1.0050
        538              0.0000   6.113430e-03         0.9806
        539              0.0000   6.066137e-03         0.9923
        540              0.0000   6.066997e-03         1.0001
        541              0.0000   6.048647e-03         0.9970
        542              0.0000   6.134174e-03         1.0141
        543              0.0000   6.160458e-03         1.0043
        544              0.0000   6.227700e-03         1.0109
        545              0.0000   6.318068e-03         1.0145
        546              0.0000   6.169826e-03         0.9765
        547              0.0000   6.221217e-03         1.0083
        548              0.0000   6.264204e-03         1.0069
        549              0.0000   6.186179e-03         0.9875
        550              0.0000   6.309242e-03         1.0199
        551              0.0000   6.353479e-03         1.0070
        552              0.0000   6.357793e-03         1.0007
        553              0.0000   6.353085e-03         0.9993
        554              0.0000   6.362545e-03         1.0015
        555              0.0000   6.307434e-03         0.9913
        556              0.0000   6.260310e-03         0.9925
        557              0.0000   6.375725e-03         1.0184
        558              0.0000   6.262039e-03         0.9822
        559              0.0000   6.339788e-03         1.0124
        560              0.0000   6.373469e-03         1.0053
        561              0.0000   6.169795e-03         0.9680
        562              0.0000   6.452991e-03         1.0459
        563              0.0000   6.528671e-03         1.0117
        564              0.0000   6.262205e-03         0.9592
        565              0.0000   6.303491e-03         1.0066
        566              0.0000   6.412179e-03         1.0172
        567              0.0000   6.303144e-03         0.9830
        568              0.0000   6.147397e-03         0.9753
        569              0.0000   6.136071e-03         0.9982
        570              0.0000   6.294733e-03         1.0259
        571              0.0000   6.012360e-03         0.9551
        572              0.0000   5.891027e-03         0.9798
        573              0.0000   5.988166e-03         1.0165
        574              0.0000   5.891136e-03         0.9838
        575              0.0000   5.977501e-03         1.0147
        576              0.0000   5.716100e-03         0.9563
        577              0.0000   5.635954e-03         0.9860
        578              0.0000   5.594633e-03         0.9927
        579              0.0000   5.477058e-03         0.9790
        580              0.0000   5.440232e-03         0.9933
        581              0.0000   5.456996e-03         1.0031
        582              0.0000   5.406281e-03         0.9907
        583              0.0000   5.364208e-03         0.9922
        584              0.0000   5.343921e-03         0.9962
        585              0.0000   5.195428e-03         0.9722
        586              0.0000   5.294369e-03         1.0190
        587              0.0000   5.331546e-03         1.0070
        588              0.0000   5.083982e-03         0.9536
        589              0.0000   5.033618e-03         0.9901
        590              0.0000   4.946860e-03         0.9828
        591              0.0000   4.898466e-03         0.9902
        592              0.0000   4.911797e-03         1.0027
        593              0.0000   4.852804e-03         0.9880
        594              0.0000   4.802611e-03         0.9897
        595              0.0000   4.875350e-03         1.0151
        596              0.0000   4.836140e-03         0.9920
        597              0.0000   4.760769e-03         0.9844
        598              0.0000   4.619881e-03         0.9704
        599              0.0000   4.380142e-03         0.9481
        600              0.0000   4.388399e-03         1.0019
        601              0.0000   4.401163e-03         1.0029
        602              0.0000   4.292870e-03         0.9754
        603              0.0000   4.236514e-03         0.9869
        604              0.0000   4.266072e-03         1.0070
        605              0.0000   4.292588e-03         1.0062
        606              0.0000   4.262486e-03         0.9930
        607              0.0000   4.249779e-03         0.9970
        608              0.0000   4.102022e-03         0.9652
        609              0.0000   4.095270e-03         0.9984
        610              0.0000   4.047164e-03         0.9883
        611              0.0000   3.910163e-03         0.9661
        612              0.0000   3.914555e-03         1.0011
        613              0.0000   3.781342e-03         0.9660
        614              0.0000   3.725926e-03         0.9853
        615              0.0000   3.809098e-03         1.0223
        616              0.0000   3.795154e-03         0.9963
        617              0.0000   3.777809e-03         0.9954
        618              0.0000   3.726416e-03         0.9864
        619              0.0000   3.552350e-03         0.9533
        620              0.0000   3.618827e-03         1.0187
        621              0.0000   3.681537e-03         1.0173
        622              0.0000   3.627929e-03         0.9854
        623              0.0000   3.571412e-03         0.9844
        624              0.0000   3.417510e-03         0.9569
        625              0.0000   3.426221e-03         1.0025
        626              0.0000   3.458219e-03         1.0093
        627              0.0000   3.438220e-03         0.9942
        628              0.0000   3.536513e-03         1.0286
        629              0.0000   3.362617e-03         0.9508
        630              0.0000   3.245816e-03         0.9653
        631              0.0000   3.292259e-03         1.0143
        632              0.0000   3.292518e-03         1.0001
        633              0.0000   3.260045e-03         0.9901
        634              0.0000   3.379410e-03         1.0366
        635              0.0000   3.318488e-03         0.9820
        636              0.0000   3.299100e-03         0.9942
        637              0.0000   3.323998e-03         1.0075
        638              0.0000   3.165984e-03         0.9525
        639              0.0000   3.162043e-03         0.9988
        640              0.0000   3.219586e-03         1.0182
        641              0.0000   3.144344e-03         0.9766
        642              0.0000   3.141328e-03         0.9990
        643              0.0000   3.223903e-03         1.0263
        644              0.0000   3.233494e-03         1.0030
        645              0.0000   3.247744e-03         1.0044
        646              0.0000   3.141434e-03         0.9673
        647              0.0000   3.092526e-03         0.9844
        648              0.0000   3.080675e-03         0.9962
        649              0.0000   3.050095e-03         0.9901
        650              0.0000   3.121482e-03         1.0234
        651              0.0000   3.205082e-03         1.0268
        652              0.0000   3.096339e-03         0.9661
        653              0.0000   3.014102e-03         0.9734
        654              0.0000   2.980229e-03         0.9888
        655              0.0000   3.016261e-03         1.0121
        656              0.0000   3.047112e-03         1.0102
        657              0.0000   3.096890e-03         1.0163
        658              0.0000   3.090594e-03         0.9980
        659              0.0000   3.049689e-03         0.9868
        660              0.0000   3.119639e-03         1.0229
        661              0.0000   3.104548e-03         0.9952
        662              0.0000   3.063124e-03         0.9867
        663              0.0000   3.071626e-03         1.0028
        664              0.0000   2.994386e-03         0.9749
        665              0.0000   3.074427e-03         1.0267
        666              0.0000   3.042785e-03         0.9897
        667              0.0000   3.002771e-03         0.9868
        668              0.0000   3.071673e-03         1.0229
        669              0.0000   3.020934e-03         0.9835
        670              0.0000   3.068377e-03         1.0157
        671              0.0000   3.032896e-03         0.9884
        672              0.0000   2.933976e-03         0.9674
        673              0.0000   2.964433e-03         1.0104
        674              0.0000   2.952689e-03         0.9960
        675              0.0000   2.885523e-03         0.9773
        676              0.0000   2.882944e-03         0.9991
        677              0.0000   2.935961e-03         1.0184
        678              0.0000   2.920453e-03         0.9947
        679              0.0000   2.858204e-03         0.9787
        680              0.0000   2.762709e-03         0.9666
        681              0.0000   2.707928e-03         0.9802
        682              0.0000   2.716042e-03         1.0030
        683              0.0000   2.721017e-03         1.0018
        684              0.0000   2.720301e-03         0.9997
        685              0.0000   2.696040e-03         0.9911
        686              0.0000   2.657746e-03         0.9858
        687              0.0000   2.664420e-03         1.0025
        688              0.0000   2.610177e-03         0.9796
        689              0.0000   2.645394e-03         1.0135
        690              0.0000   2.699657e-03         1.0205
        691              0.0000   2.609284e-03         0.9665
        692              0.0000   2.629226e-03         1.0076
        693              0.0000   2.611216e-03         0.9932
        694              0.0000   2.630112e-03         1.0072
        695              0.0000   2.653442e-03         1.0089
        696              0.0000   2.538814e-03         0.9568
        697              0.0000   2.502645e-03         0.9858
        698              0.0000   2.504065e-03         1.0006
        699              0.0000   2.687520e-03         1.0733
        700              0.0000   2.551698e-03         0.9495
        701              0.0000   2.461522e-03         0.9647
        702              0.0000   2.531003e-03         1.0282
        703              0.0000   2.430953e-03         0.9605
        704              0.0000   2.430052e-03         0.9996
        705              0.0000   2.504488e-03         1.0306
        706              0.0000   2.466978e-03         0.9850
        707              0.0000   2.456033e-03         0.9956
        708              0.0000   2.491324e-03         1.0144
        709              0.0000   2.426681e-03         0.9741
        710              0.0000   2.391890e-03         0.9857
        711              0.0000   2.478016e-03         1.0360
        712              0.0000   2.511699e-03         1.0136
        713              0.0000   2.449575e-03         0.9753
        714              0.0000   2.481302e-03         1.0130
        715              0.0000   2.505996e-03         1.0100
        716              0.0000   2.501104e-03         0.9980
        717              0.0000   2.451500e-03         0.9802
        718              0.0000   2.477735e-03         1.0107
        719              0.0000   2.442478e-03         0.9858
        720              0.0000   2.383028e-03         0.9757
        721              0.0000   2.382710e-03         0.9999
        722              0.0000   2.370649e-03         0.9949
        723              0.0000   2.415416e-03         1.0189
        724              0.0000   2.438737e-03         1.0097
        725              0.0000   2.407252e-03         0.9871
        726              0.0000   2.346697e-03         0.9748
        727              0.0000   2.321115e-03         0.9891
        728              0.0000   2.333413e-03         1.0053
        729              0.0000   2.351854e-03         1.0079
        730              0.0000   2.409359e-03         1.0245
        731              0.0000   2.344550e-03         0.9731
        732              0.0000   2.293627e-03         0.9783
        733              0.0000   2.371450e-03         1.0339
        734              0.0000   2.287999e-03         0.9648
        735              0.0000   2.313765e-03         1.0113
        736              0.0000   2.260754e-03         0.9771
        737              0.0000   2.188349e-03         0.9680
        738              0.0000   2.238853e-03         1.0231
        739              0.0000   2.291927e-03         1.0237
        740              0.0000   2.258121e-03         0.9853
        741              0.0000   2.213533e-03         0.9803
        742              0.0000   2.192415e-03         0.9905
        743              0.0000   2.159613e-03         0.9850
        744              0.0000   2.152427e-03         0.9967
        745              0.0000   2.170189e-03         1.0083
        746              0.0000   2.223195e-03         1.0244
        747              0.0000   2.307541e-03         1.0379
        748              0.0000   2.181823e-03         0.9455
        749              0.0000   2.163917e-03         0.9918
        750              0.0000   2.167458e-03         1.0016
        751              0.0000   2.156657e-03         0.9950
        752              0.0000   2.125397e-03         0.9855
        753              0.0000   2.142835e-03         1.0082
        754              0.0000   2.217114e-03         1.0347
        755              0.0000   2.107002e-03         0.9503
        756              0.0000   2.074550e-03         0.9846
        757              0.0000   2.047449e-03         0.9869
        758              0.0000   2.003972e-03         0.9788
        759              0.0000   2.014668e-03         1.0053
        760              0.0000   1.975443e-03         0.9805
        761              0.0000   1.945485e-03         0.9848
        762              0.0000   1.952831e-03         1.0038
        763              0.0000   1.937567e-03         0.9922
        764              0.0000   1.894101e-03         0.9776
        765              0.0000   1.868663e-03         0.9866
        766              0.0000   1.876975e-03         1.0044
        767              0.0000   1.855223e-03         0.9884
        768              0.0000   1.856640e-03         1.0008
        769              0.0000   1.873003e-03         1.0088
        770              0.0000   1.825525e-03         0.9747
        771              0.0000   1.772884e-03         0.9712
        772              0.0000   1.791424e-03         1.0105
        773              0.0000   1.761388e-03         0.9832
        774              0.0000   1.754499e-03         0.9961
        775              0.0000   1.749571e-03         0.9972
        776              0.0000   1.732596e-03         0.9903
        777              0.0000   1.727874e-03         0.9973
        778              0.0000   1.732001e-03         1.0024
        779              0.0000   1.727282e-03         0.9973
        780              0.0000   1.700106e-03         0.9843
        781              0.0000   1.703887e-03         1.0022
        782              0.0000   1.680089e-03         0.9860
        783              0.0000   1.628074e-03         0.9690
        784              0.0000   1.628501e-03         1.0003
        785              0.0000   1.636634e-03         1.0050
        786              0.0000   1.628870e-03         0.9953
        787              0.0000   1.641730e-03         1.0079
        788              0.0000   1.625949e-03         0.9904
        789              0.0000   1.581212e-03         0.9725
        790              0.0000   1.532030e-03         0.9689
        791              0.0000   1.519634e-03         0.9919
        792              0.0000   1.535160e-03         1.0102
        793              0.0000   1.550009e-03         1.0097
        794              0.0000   1.563274e-03         1.0086
        795              0.0000   1.493288e-03         0.9552
        796              0.0000   1.459549e-03         0.9774
        797              0.0000   1.509726e-03         1.0344
        798              0.0000   1.470356e-03         0.9739
        799              0.0000   1.436431e-03         0.9769
        800              0.0000   1.411141e-03         0.9824
        801              0.0000   1.403195e-03         0.9944
        802              0.0000   1.402421e-03         0.9994
        803              0.0000   1.385873e-03         0.9882
        804              0.0000   1.353841e-03         0.9769
        805              0.0000   1.340951e-03         0.9905
        806              0.0000   1.341311e-03         1.0003
        807              0.0000   1.349036e-03         1.0058
        808              0.0000   1.350563e-03         1.0011
        809              0.0000   1.285433e-03         0.9518
        810              0.0000   1.302456e-03         1.0132
        811              0.0000   1.317619e-03         1.0116
        812              0.0000   1.323645e-03         1.0046
        813              0.0000   1.322994e-03         0.9995
        814              0.0000   1.356128e-03         1.0250
        815              0.0000   1.317021e-03         0.9712
        816              0.0000   1.286394e-03         0.9767
        817              0.0000   1.293315e-03         1.0054
        818              0.0000   1.301398e-03         1.0062
        819              0.0000   1.347228e-03         1.0352
        820              0.0000   1.294844e-03         0.9611
        821              0.0000   1.260692e-03         0.9736
        822              0.0000   1.269896e-03         1.0073
        823              0.0000   1.234638e-03         0.9722
        824              0.0000   1.231516e-03         0.9975
        825              0.0000   1.250195e-03         1.0152
        826              0.0000   1.237134e-03         0.9896
        827              0.0000   1.258198e-03         1.0170
        828              0.0000   1.279474e-03         1.0169
        829              0.0000   1.274285e-03         0.9959
        830              0.0000   1.252230e-03         0.9827
        831              0.0000   1.216491e-03         0.9715
        832              0.0000   1.204917e-03         0.9905
        833              0.0000   1.167398e-03         0.9689
        834              0.0000   1.161045e-03         0.9946
        835              0.0000   1.173960e-03         1.0111
        836              0.0000   1.182561e-03         1.0073
        837              0.0000   1.192346e-03         1.0083
        838              0.0000   1.148196e-03         0.9630
        839              0.0000   1.092183e-03         0.9512
        840              0.0000   1.058793e-03         0.9694
        841              0.0000   1.045892e-03         0.9878
        842              0.0000   1.038956e-03         0.9934
        843              0.0000   1.037247e-03         0.9984
        844              0.0000   1.042432e-03         1.0050
        845              0.0000   1.049370e-03         1.0067
        846              0.0000   1.064085e-03         1.0140
        847              0.0000   1.021784e-03         0.9602
        848              0.0000   9.960429e-04         0.9748
        849              0.0000   9.829912e-04         0.9869
        850              0.0000   9.908452e-04         1.0080
        851              0.0000   1.005408e-03         1.0147
        852              0.0000   9.850166e-04         0.9797
        853              0.0000   9.731450e-04         0.9879
        854              0.0000   9.615552e-04         0.9881
        855              0.0000   9.421366e-04         0.9798
        856              0.0000   9.616912e-04         1.0208
        857              0.0000   9.832599e-04         1.0224
        858              0.0000   9.875020e-04         1.0043
        859              0.0000   9.857665e-04         0.9982
        860              0.0000   9.712259e-04         0.9852
        861              0.0000   9.714164e-04         1.0002
        862              0.0000   9.580410e-04         0.9862
        863              0.0000   9.725060e-04         1.0151
        864              0.0000   9.726826e-04         1.0002
        865              0.0000   9.394663e-04         0.9659
        866              0.0000   9.413713e-04         1.0020
        867              0.0000   9.396075e-04         0.9981
        868              0.0000   9.409370e-04         1.0014
        869              0.0000   9.498303e-04         1.0095
        870              0.0000   9.261593e-04         0.9751
        871              0.0000   9.199120e-04         0.9933
        872              0.0000   9.200373e-04         1.0001
        873              0.0000   9.041397e-04         0.9827
        874              0.0000   9.123642e-04         1.0091
        875              0.0000   9.192381e-04         1.0075
        876              0.0000   9.188214e-04         0.9995
        877              0.0000   9.000381e-04         0.9796
        878              0.0000   8.806142e-04         0.9784
        879              0.0000   8.855791e-04         1.0056
        880              0.0000   8.676648e-04         0.9798
        881              0.0000   8.677054e-04         1.0000
        882              0.0000   8.807259e-04         1.0150
        883              0.0000   8.676048e-04         0.9851
        884              0.0000   8.676534e-04         1.0001
        885              0.0000   8.309741e-04         0.9577
        886              0.0000   8.163647e-04         0.9824
        887              0.0000   8.268292e-04         1.0128
        888              0.0000   8.249113e-04         0.9977
        889              0.0000   8.431913e-04         1.0222
        890              0.0000   8.093790e-04         0.9599
        891              0.0000   8.053845e-04         0.9951
        892              0.0000   8.313124e-04         1.0322
        893              0.0000   7.973871e-04         0.9592
        894              0.0000   7.883449e-04         0.9887
        895              0.0000   7.711701e-04         0.9782
        896              0.0000   7.718840e-04         1.0009
        897              0.0000   7.925186e-04         1.0267
        898              0.0000   8.114793e-04         1.0239
        899              0.0000   8.119894e-04         1.0006
        900              0.0000   7.745538e-04         0.9539
        901              0.0000   7.748123e-04         1.0003
        902              0.0000   7.678743e-04         0.9910
        903              0.0000   7.640441e-04         0.9950
        904              0.0000   7.755048e-04         1.0150
        905              0.0000   7.891542e-04         1.0176
        906              0.0000   7.742804e-04         0.9812
        907              0.0000   7.525522e-04         0.9719
        908              0.0000   7.566220e-04         1.0054
        909              0.0000   7.469665e-04         0.9872
        910              0.0000   7.290903e-04         0.9761
        911              0.0000   7.522938e-04         1.0318
        912              0.0000   7.442352e-04         0.9893
        913              0.0000   7.108979e-04         0.9552
        914              0.0000   7.095701e-04         0.9981
        915              0.0000   7.070365e-04         0.9964
        916              0.0000   7.088656e-04         1.0026
        917              0.0000   6.982154e-04         0.9850
        918              0.0000   6.753450e-04         0.9672
        919              0.0000   6.675135e-04         0.9884
        920              0.0000   6.671596e-04         0.9995
        921              0.0000   6.708132e-04         1.0055
        922              0.0000   6.444531e-04         0.9607
        923              0.0000   6.226790e-04         0.9662
        924              0.0000   6.230278e-04         1.0006
        925              0.0000   6.163252e-04         0.9892
        926              0.0000   6.022176e-04         0.9771
        927              0.0000   5.991566e-04         0.9949
        928              0.0000   5.912835e-04         0.9869
        929              0.0000   5.831130e-04         0.9862
        930              0.0000   5.700193e-04         0.9775
        931              0.0000   5.571408e-04         0.9774
        932              0.0000   5.691536e-04         1.0216
        933              0.0000   5.579242e-04         0.9803
        934              0.0000   5.528541e-04         0.9909
        935              0.0000   5.490729e-04         0.9932
        936              0.0000   5.333079e-04         0.9713
        937              0.0000   5.326629e-04         0.9988
        938              0.0000   5.286821e-04         0.9925
        939              0.0000   5.300768e-04         1.0026
        940              0.0000   5.217623e-04         0.9843
        941              0.0000   5.058423e-04         0.9695
        942              0.0000   4.945977e-04         0.9778
        943              0.0000   4.943199e-04         0.9994
        944              0.0000   5.014849e-04         1.0145
        945              0.0000   4.948979e-04         0.9869
        946              0.0000   4.834331e-04         0.9768
        947              0.0000   4.703484e-04         0.9729
        948              0.0000   4.760264e-04         1.0121
        949              0.0000   4.698934e-04         0.9871
        950              0.0000   4.603654e-04         0.9797
        951              0.0000   4.505246e-04         0.9786
        952              0.0000   4.449649e-04         0.9877
        953              0.0000   4.477592e-04         1.0063
        954              0.0000   4.486911e-04         1.0021
        955              0.0000   4.348044e-04         0.9691
        956              0.0000   4.218062e-04         0.9701
        957              0.0000   4.101086e-04         0.9723
        958              0.0000   4.066455e-04         0.9916
        959              0.0000   4.043486e-04         0.9944
        960              0.0000   3.920222e-04         0.9695
        961              0.0000   3.847938e-04         0.9816
        962              0.0000   3.794788e-04         0.9862
        963              0.0000   3.799792e-04         1.0013
        964              0.0000   3.839192e-04         1.0104
        965              0.0000   3.715595e-04         0.9678
        966              0.0000   3.647828e-04         0.9818
        967              0.0000   3.574942e-04         0.9800
        968              0.0000   3.566596e-04         0.9977
        969              0.0000   3.477963e-04         0.9751
        970              0.0000   3.457679e-04         0.9942
        971              0.0000   3.517328e-04         1.0173
        972              0.0000   3.450002e-04         0.9809
        973              0.0000   3.401140e-04         0.9858
        974              0.0000   3.331621e-04         0.9796
        975              0.0000   3.234024e-04         0.9707
        976              0.0000   3.258192e-04         1.0075
        977              0.0000   3.350614e-04         1.0284
        978              0.0000   3.334387e-04         0.9952
        979              0.0000   3.293452e-04         0.9877
        980              0.0000   3.211299e-04         0.9751
        981              0.0000   3.210761e-04         0.9998
        982              0.0000   3.164181e-04         0.9855
        983              0.0000   3.050857e-04         0.9642
        984              0.0000   3.012009e-04         0.9873
        985              0.0000   3.024822e-04         1.0043
        986              0.0000   3.026344e-04         1.0005
        987              0.0000   3.086870e-04         1.0200
        988              0.0000   3.049964e-04         0.9880
        989              0.0000   2.873506e-04         0.9421
        990              0.0000   2.834352e-04         0.9864
        991              0.0000   2.808062e-04         0.9907
        992              0.0000   2.793854e-04         0.9949
        993              0.0000   2.865596e-04         1.0257
        994              0.0000   2.866176e-04         1.0002
        995              0.0000   2.788491e-04         0.9729
        996              0.0000   2.758293e-04         0.9892
        997              0.0000   2.763111e-04         1.0017
        998              0.0000   2.682861e-04         0.9710
        999              0.0000   2.601792e-04         0.9698
       1000              0.0000   2.596169e-04         0.9978
       1001              0.0000   2.616959e-04         1.0080
       1002              0.0000   2.535893e-04         0.9690
       1003              0.0000   2.456781e-04         0.9688
       1004              0.0000   2.442945e-04         0.9944
       1005              0.0000   2.418638e-04         0.9900
       1006              0.0000   2.453883e-04         1.0146
       1007              0.0000   2.476002e-04         1.0090
       1008              0.0000   2.420985e-04         0.9778
       1009              0.0000   2.323620e-04         0.9598
       1010              0.0000   2.233616e-04         0.9613
       1011              0.0000   2.289190e-04         1.0249
       1012              0.0000   2.321657e-04         1.0142
       1013              0.0000   2.274925e-04         0.9799
       1014              0.0000   2.191199e-04         0.9632
       1015              0.0000   2.175438e-04         0.9928
       1016              0.0000   2.220332e-04         1.0206
       1017              0.0000   2.172046e-04         0.9783
       1018              0.0000   2.097659e-04         0.9658
       1019              0.0000   2.036157e-04         0.9707
       1020              0.0000   1.994712e-04         0.9796
       1021              0.0000   1.959702e-04         0.9824
       1022              0.0000   1.930542e-04         0.9851
       1023              0.0000   1.909361e-04         0.9890
       1024              0.0000   1.903897e-04         0.9971
       1025              0.0000   1.917316e-04         1.0070
       1026              0.0000   1.910562e-04         0.9965
       1027              0.0000   1.833002e-04         0.9594
       1028              0.0000   1.776087e-04         0.9689
       1029              0.0000   1.754119e-04         0.9876
       1030              0.0000   1.745686e-04         0.9952
       1031              0.0000   1.718461e-04         0.9844
       1032              0.0000   1.664330e-04         0.9685
       1033              0.0000   1.644253e-04         0.9879
       1034              0.0000   1.606550e-04         0.9771
       1035              0.0000   1.561615e-04         0.9720
       1036              0.0000   1.522687e-04         0.9751
       1037              0.0000   1.486872e-04         0.9765
       1038              0.0000   1.492920e-04         1.0041
       1039              0.0000   1.525888e-04         1.0221
       1040              0.0000   1.458966e-04         0.9561
       1041              0.0000   1.426591e-04         0.9778
       1042              0.0000   1.399706e-04         0.9812
       1043              0.0000   1.379055e-04         0.9852
       1044              0.0000   1.389959e-04         1.0079
       1045              0.0000   1.386503e-04         0.9975
       1046              0.0000   1.380247e-04         0.9955
       1047              0.0000   1.357530e-04         0.9835
       1048              0.0000   1.356600e-04         0.9993
       1049              0.0000   1.340695e-04         0.9883
       1050              0.0000   1.305877e-04         0.9740
       1051              0.0000   1.261571e-04         0.9661
       1052              0.0000   1.238009e-04         0.9813
       1053              0.0000   1.238562e-04         1.0004
       1054              0.0000   1.280320e-04         1.0337
       1055              0.0000   1.288540e-04         1.0064
       1056              0.0000   1.192187e-04         0.9252
       1057              0.0000   1.171484e-04         0.9826
       1058              0.0000   1.177575e-04         1.0052
       1059              0.0000   1.160407e-04         0.9854
       1060              0.0000   1.192555e-04         1.0277
       1061              0.0000   1.145663e-04         0.9607
       1062              0.0000   1.119191e-04         0.9769
       1063              0.0000   1.113072e-04         0.9945
       1064              0.0000   1.066629e-04         0.9583
       1065              0.0000   1.082836e-04         1.0152
       1066              0.0000   1.094147e-04         1.0104
       1067              0.0000   1.078406e-04         0.9856
       1068              0.0000   1.059389e-04         0.9824
       1069              0.0000   1.025855e-04         0.9683
       1070              0.0000   1.065688e-04         1.0388
       1071              0.0000   1.038323e-04         0.9743
       1072              0.0000   1.005078e-04         0.9680
       1073              0.0000   9.960116e-05         0.9910
       1074              0.0000   9.555465e-05         0.9594
       1075              0.0000   9.389676e-05         0.9826
       1076              0.0000   9.329190e-05         0.9936
       1077              0.0000   9.168408e-05         0.9828
       1078              0.0000   9.045320e-05         0.9866
       1079              0.0000   9.012966e-05         0.9964
       1080              0.0000   8.915993e-05         0.9892
       1081              0.0000   8.807006e-05         0.9878
       1082              0.0000   8.689666e-05         0.9867
       1083              0.0000   8.928345e-05         1.0275
       1084              0.0000   8.881869e-05         0.9948
       1085              0.0000   8.305333e-05         0.9351
       1086              0.0000   8.164746e-05         0.9831
       1087              0.0000   8.361787e-05         1.0241
       1088              0.0000   8.482257e-05         1.0144
       1089              0.0000   8.102409e-05         0.9552
       1090              0.0000   7.804316e-05         0.9632
       1091              0.0000   7.777590e-05         0.9966
       1092              0.0000   7.698070e-05         0.9898
       1093              0.0000   7.617721e-05         0.9896
       1094              0.0000   7.785015e-05         1.0220
       1095              0.0000   7.908359e-05         1.0158
       1096              0.0000   7.586426e-05         0.9593
       1097              0.0000   7.311162e-05         0.9637
       1098              0.0000   7.077980e-05         0.9681
       1099              0.0000   7.057252e-05         0.9971
       1100              0.0000   7.018112e-05         0.9945
       1101              0.0000   6.975981e-05         0.9940
       1102              0.0000   7.069918e-05         1.0135
       1103              0.0000   6.917076e-05         0.9784
       1104              0.0000   6.798418e-05         0.9828
       1105              0.0000   6.724114e-05         0.9891
       1106              0.0000   6.722599e-05         0.9998
       1107              0.0000   6.703655e-05         0.9972
       1108              0.0000   6.766579e-05         1.0094
       1109              0.0000   6.728036e-05         0.9943
       1110              0.0000   6.337604e-05         0.9420
       1111              0.0000   6.185643e-05         0.9760
       1112              0.0000   6.180634e-05         0.9992
       1113              0.0000   6.237878e-05         1.0093
       1114              0.0000   6.059123e-05         0.9713
       1115              0.0000   5.896654e-05         0.9732
       1116              0.0000   5.900299e-05         1.0006
       1117              0.0000   5.988532e-05         1.0150
       1118              0.0000   5.821689e-05         0.9721
       1119              0.0000   5.568219e-05         0.9565
       1120              0.0000   5.459460e-05         0.9805
       1121              0.0000   5.404070e-05         0.9899
       1122              0.0000   5.327976e-05         0.9859
       1123              0.0000   5.329041e-05         1.0002
       1124              0.0000   5.310982e-05         0.9966
       1125              0.0000   5.209220e-05         0.9808
       1126              0.0000   5.205710e-05         0.9993
       1127              0.0000   5.152420e-05         0.9898
       1128              0.0000   5.079353e-05         0.9858
       1129              0.0000   4.902484e-05         0.9652
       1130              0.0000   4.714705e-05         0.9617
       1131              0.0000   4.671814e-05         0.9909
       1132              0.0000   4.722716e-05         1.0109
       1133              0.0000   4.672086e-05         0.9893
       1134              0.0000   4.591729e-05         0.9828
       1135              0.0000   4.543921e-05         0.9896
       1136              0.0000   4.455681e-05         0.9806
       1137              0.0000   4.512239e-05         1.0127
       1138              0.0000   4.406280e-05         0.9765
       1139              0.0000   4.304828e-05         0.9770
       1140              0.0000   4.228005e-05         0.9822
       1141              0.0000   4.117294e-05         0.9738
       1142              0.0000   4.094077e-05         0.9944
       1143              0.0000   4.121272e-05         1.0066
       1144              0.0000   4.008860e-05         0.9727
       1145              0.0000   3.926912e-05         0.9796
       1146              0.0000   3.884349e-05         0.9892
       1147              0.0000   3.774906e-05         0.9718
       1148              0.0000   3.682124e-05         0.9754
       1149              0.0000   3.645388e-05         0.9900
       1150              0.0000   3.626818e-05         0.9949
       1151              0.0000   3.691003e-05         1.0177
       1152              0.0000   3.568361e-05         0.9668
       1153              0.0000   3.417539e-05         0.9577
       1154              0.0000   3.359662e-05         0.9831
       1155              0.0000   3.366826e-05         1.0021
       1156              0.0000   3.365647e-05         0.9996
       1157              0.0000   3.293674e-05         0.9786
       1158              0.0000   3.176295e-05         0.9644
       1159              0.0000   3.052760e-05         0.9611
       1160              0.0000   3.032254e-05         0.9933
       1161              0.0000   2.970485e-05         0.9796
       1162              0.0000   2.880702e-05         0.9698
       1163              0.0000   2.869126e-05         0.9960
       1164              0.0000   2.866974e-05         0.9992
       1165              0.0000   2.925789e-05         1.0205
       1166              0.0000   2.978360e-05         1.0180
       1167              0.0000   2.862327e-05         0.9610
       1168              0.0000   2.773773e-05         0.9691
       1169              0.0000   2.713071e-05         0.9781
       1170              0.0000   2.739368e-05         1.0097
       1171              0.0000   2.752526e-05         1.0048
       1172              0.0000   2.726573e-05         0.9906
       1173              0.0000   2.669564e-05         0.9791
       1174              0.0000   2.588392e-05         0.9696
       1175              0.0000   2.542371e-05         0.9822
       1176              0.0000   2.491846e-05         0.9801
       1177              0.0000   2.492559e-05         1.0003
       1178              0.0000   2.416940e-05         0.9697
       1179              0.0000   2.377181e-05         0.9835
       1180              0.0000   2.363671e-05         0.9943
       1181              0.0000   2.398205e-05         1.0146
       1182              0.0000   2.405632e-05         1.0031
       1183              0.0000   2.271367e-05         0.9442
       1184              0.0000   2.249473e-05         0.9904
       1185              0.0000   2.231221e-05         0.9919
       1186              0.0000   2.193788e-05         0.9832
       1187              0.0000   2.178802e-05         0.9932
       1188              0.0000   2.121793e-05         0.9738
       1189              0.0000   2.104937e-05         0.9921
       1190              0.0000   2.149159e-05         1.0210
       1191              0.0000   2.074685e-05         0.9653
       1192              0.0000   2.000348e-05         0.9642
       1193              0.0000   2.010386e-05         1.0050
       1194              0.0000   1.991699e-05         0.9907
       1195              0.0000   1.964748e-05         0.9865
       1196              0.0000   1.946989e-05         0.9910
       1197              0.0000   1.925539e-05         0.9890
       1198              0.0000   1.917586e-05         0.9959
       1199              0.0000   1.894290e-05         0.9879
       1200              0.0000   1.899356e-05         1.0027
       1201              0.0000   1.884265e-05         0.9921
       1202              0.0000   1.807620e-05         0.9593
       1203              0.0000   1.796752e-05         0.9940
       1204              0.0000   1.803827e-05         1.0039
       1205              0.0000   1.774020e-05         0.9835
       1206              0.0000   1.721043e-05         0.9701
       1207              0.0000   1.674415e-05         0.9729
       1208              0.0000   1.658830e-05         0.9907
       1209              0.0000   1.667041e-05         1.0050
       1210              0.0000   1.703125e-05         1.0216
       1211              0.0000   1.649411e-05         0.9685
       1212              0.0000   1.617734e-05         0.9808
       1213              0.0000   1.657556e-05         1.0246
       1214              0.0000   1.634708e-05         0.9862
       1215              0.0000   1.587665e-05         0.9712
       1216              0.0000   1.564958e-05         0.9857
       1217              0.0000   1.550977e-05         0.9911
       1218              0.0000   1.568440e-05         1.0113
       1219              0.0000   1.520927e-05         0.9697
       1220              0.0000   1.467077e-05         0.9646
       1221              0.0000   1.451754e-05         0.9896
       1222              0.0000   1.433151e-05         0.9872
       1223              0.0000   1.443223e-05         1.0070
       1224              0.0000   1.450418e-05         1.0050
       1225              0.0000   1.426128e-05         0.9833
       1226              0.0000   1.393596e-05         0.9772
       1227              0.0000   1.368879e-05         0.9823
       1228              0.0000   1.374039e-05         1.0038
       1229              0.0000   1.369593e-05         0.9968
       1230              0.0000   1.353448e-05         0.9882
     ----------------------------------------------------------------------
     Total Iterations: 1231
     Avg Convergence Rate:               0.9888
     Final Residual:           1.353448e-05
     Total Reduction in Residual:      9.996766e-07
     Maximum Memory Usage:                0.000 GB
     ----------------------------------------------------------------------

Total Time: 5.72287 setup: 0.000170048 s solve: 5.7227 s solve(per iteration): 0.00464882 s

_If I use GMRES_AMG_D2.json it took 16s_ AMGX version 2.5.0 Built on Apr 25 2024, 13:20:11 Compiled with CUDA Runtime 12.4, using CUDA driver 12.4 AMG Grid: Number of Levels: 9 LVL ROWS NNZ PARTS SPRSTY Mem (GB)

       0(D)      1507876         113305856      1  4.98e-05            1.3
       1(D)       281692          31103364      1  0.000392          0.706
       2(D)        39859           4883517      1   0.00307          0.111
       3(D)         4793            328295      1    0.0143        0.00753
       4(D)          762             35364      1    0.0609       0.000822
       5(D)          198              7160      1     0.183       0.000168
       6(D)           60              1622      1     0.451       3.88e-05
       7(D)           18               254      1     0.784       6.45e-06
       8(D)            6                36      1         1       9.98e-07
     ----------------------------------------------------------------------
     Grid Complexity: 1.21712
     Operator Complexity: 1.3209
     Total Memory Usage: 2.12977 GB
     ----------------------------------------------------------------------
       iter      Mem Usage (GB)       residual           rate
     ----------------------------------------------------------------------
        Ini             4.50732   1.353886e+01
          0             4.50732   9.552636e+00         0.7056
          1              4.5073   6.826744e+00         0.7146
          2              4.5073   5.451789e+00         0.7986
          3              4.5073   4.193551e+00         0.7692
          4              4.5073   3.318682e+00         0.7914
          5              4.5073   2.696656e+00         0.8126
          6              4.5073   2.253779e+00         0.8358
          7              4.5073   1.897404e+00         0.8419
          8              4.5073   1.605763e+00         0.8463
          9              4.5073   1.371794e+00         0.8543
         10              4.5073   1.280317e+00         0.9333
         11              4.5073   1.165342e+00         0.9102
         12              4.5073   1.044937e+00         0.8967
         13              4.5073   9.311771e-01         0.8911
         14              4.5073   8.317691e-01         0.8932
         15              4.5073   7.104873e-01         0.8542
         16              4.5073   6.342603e-01         0.8927
         17              4.5073   5.645147e-01         0.8900
         18              4.5073   5.062215e-01         0.8967
         19              4.5073   4.567174e-01         0.9022
         20              4.5073   4.388230e-01         0.9608
         21              4.5073   4.211402e-01         0.9597
         22              4.5073   3.973845e-01         0.9436
         23              4.5073   3.710982e-01         0.9339
         24              4.5073   3.424965e-01         0.9229
         25              4.5073   3.155958e-01         0.9215
         26              4.5073   2.966112e-01         0.9398
         27              4.5073   2.795076e-01         0.9423
         28              4.5073   2.635899e-01         0.9431
         29              4.5073   2.461252e-01         0.9337
         30              4.5073   2.365946e-01         0.9613
         31              4.5073   2.264181e-01         0.9570
         32              4.5073   2.144454e-01         0.9471
         33              4.5073   2.034822e-01         0.9489
         34              4.5073   1.939666e-01         0.9532
         35              4.5073   1.830933e-01         0.9439
         36              4.5073   1.759580e-01         0.9610
         37              4.5073   1.689407e-01         0.9601
         38              4.5073   1.625273e-01         0.9620
         39              4.5073   1.567911e-01         0.9647
         40              4.5073   1.523330e-01         0.9716
         41              4.5073   1.482169e-01         0.9730
         42              4.5073   1.430960e-01         0.9655
         43              4.5073   1.379123e-01         0.9638
         44              4.5073   1.318838e-01         0.9563
         45              4.5073   1.262892e-01         0.9576
         46              4.5073   1.216729e-01         0.9634
         47              4.5073   1.172642e-01         0.9638
         48              4.5073   1.136271e-01         0.9690
         49              4.5073   1.098468e-01         0.9667
         50              4.5073   1.073779e-01         0.9775
         51              4.5073   1.044496e-01         0.9727
         52              4.5073   1.007739e-01         0.9648
         53              4.5073   9.737754e-02         0.9663
         54              4.5073   9.470861e-02         0.9726
         55              4.5073   9.166303e-02         0.9678
         56              4.5073   8.969872e-02         0.9786
         57              4.5073   8.761253e-02         0.9767
         58              4.5073   8.539046e-02         0.9746
         59              4.5073   8.346770e-02         0.9775
         60              4.5073   8.175577e-02         0.9795
         61              4.5073   8.011439e-02         0.9799
         62              4.5073   7.851187e-02         0.9800
         63              4.5073   7.699035e-02         0.9806
         64              4.5073   7.499845e-02         0.9741
         65              4.5073   7.315360e-02         0.9754
         66              4.5073   7.119191e-02         0.9732
         67              4.5073   6.927591e-02         0.9731
         68              4.5073   6.783679e-02         0.9792
         69              4.5073   6.647737e-02         0.9800
         70              4.5073   6.545250e-02         0.9846
         71              4.5073   6.426370e-02         0.9818
         72              4.5073   6.257041e-02         0.9737
         73              4.5073   6.100275e-02         0.9749
         74              4.5073   5.987852e-02         0.9816
         75              4.5073   5.862508e-02         0.9791
         76              4.5073   5.780494e-02         0.9860
         77              4.5073   5.687329e-02         0.9839
         78              4.5073   5.563071e-02         0.9782
         79              4.5073   5.456461e-02         0.9808
         80              4.5073   5.351356e-02         0.9807
         81              4.5073   5.255514e-02         0.9821
         82              4.5073   5.177287e-02         0.9851
         83              4.5073   5.106999e-02         0.9864
         84              4.5073   5.012493e-02         0.9815
         85              4.5073   4.921204e-02         0.9818
         86              4.5073   4.799043e-02         0.9752
         87              4.5073   4.670630e-02         0.9732
         88              4.5073   4.583710e-02         0.9814
         89              4.5073   4.502878e-02         0.9824
         90              4.5073   4.445102e-02         0.9872
         91              4.5073   4.367765e-02         0.9826
         92              4.5073   4.254507e-02         0.9741
         93              4.5073   4.147386e-02         0.9748
         94              4.5073   4.078047e-02         0.9833
         95              4.5073   4.004918e-02         0.9821
         96              4.5073   3.958986e-02         0.9885
         97              4.5073   3.904670e-02         0.9863
         98              4.5073   3.819285e-02         0.9781
         99              4.5073   3.743150e-02         0.9801
        100              4.5073   3.671366e-02         0.9808
        101              4.5073   3.597912e-02         0.9800
        102              4.5073   3.552583e-02         0.9874
        103              4.5073   3.510992e-02         0.9883
        104              4.5073   3.452025e-02         0.9832
        105              4.5073   3.394837e-02         0.9834
        106              4.5073   3.301806e-02         0.9726
        107              4.5073   3.206440e-02         0.9711
        108              4.5073   3.147588e-02         0.9816
        109              4.5073   3.095063e-02         0.9833
        110              4.5073   3.051699e-02         0.9860
        111              4.5073   3.003388e-02         0.9842
        112              4.5073   2.918968e-02         0.9719
        113              4.5073   2.837800e-02         0.9722
        114              4.5073   2.792080e-02         0.9839
        115              4.5073   2.745213e-02         0.9832
        116              4.5073   2.715536e-02         0.9892
        117              4.5073   2.684906e-02         0.9887
        118              4.5073   2.618969e-02         0.9754
        119              4.5073   2.561850e-02         0.9782
        120              4.5073   2.506056e-02         0.9782
        121              4.5073   2.451296e-02         0.9781
        122              4.5073   2.423323e-02         0.9886
        123              4.5073   2.397209e-02         0.9892
        124              4.5073   2.358096e-02         0.9837
        125              4.5073   2.317335e-02         0.9827
        126              4.5073   2.252357e-02         0.9720
        127              4.5073   2.182169e-02         0.9688
        128              4.5073   2.142175e-02         0.9817
        129              4.5073   2.106657e-02         0.9834
        130              4.5073   2.076983e-02         0.9859
        131              4.5073   2.043719e-02         0.9840
        132              4.5073   1.982875e-02         0.9702
        133              4.5073   1.925887e-02         0.9713
        134              4.5073   1.892543e-02         0.9827
        135              4.5073   1.861500e-02         0.9836
        136              4.5073   1.841600e-02         0.9893
        137              4.5073   1.822721e-02         0.9897
        138              4.5073   1.774987e-02         0.9738
        139              4.5073   1.723753e-02         0.9711
        140              4.5073   1.675373e-02         0.9719
        141              4.5073   1.636719e-02         0.9769
        142              4.5073   1.617550e-02         0.9883
        143              4.5073   1.600921e-02         0.9897
        144              4.5073   1.578544e-02         0.9860
        145              4.5073   1.549496e-02         0.9816
        146              4.5073   1.508306e-02         0.9734
        147              4.5073   1.456481e-02         0.9656
        148              4.5073   1.427043e-02         0.9798
        149              4.5073   1.402647e-02         0.9829
        150              4.5073   1.384524e-02         0.9871
        151              4.5073   1.359538e-02         0.9820
        152              4.5073   1.315202e-02         0.9674
        153              4.5073   1.278672e-02         0.9722
        154              4.5073   1.253887e-02         0.9806
        155              4.5073   1.235779e-02         0.9856
        156              4.5073   1.223345e-02         0.9899
        157              4.5073   1.210488e-02         0.9895
        158              4.5073   1.180894e-02         0.9756
        159              4.5073   1.147988e-02         0.9721
        160              4.5073   1.116957e-02         0.9730
        161              4.5073   1.092455e-02         0.9781
        162              4.5073   1.079219e-02         0.9879
        163              4.5073   1.069048e-02         0.9906
        164              4.5073   1.054792e-02         0.9867
        165              4.5073   1.033906e-02         0.9802
        166              4.5073   1.006528e-02         0.9735
        167              4.5073   9.688645e-03         0.9626
        168              4.5073   9.485966e-03         0.9791
        169              4.5073   9.321227e-03         0.9826
        170              4.5073   9.195419e-03         0.9865
        171              4.5073   9.021929e-03         0.9811
        172              4.5073   8.718005e-03         0.9663
        173              4.5073   8.466200e-03         0.9711
        174              4.5073   8.285718e-03         0.9787
        175              4.5073   8.173952e-03         0.9865
        176              4.5073   8.095984e-03         0.9905
        177              4.5073   8.004101e-03         0.9887
        178              4.5073   7.829951e-03         0.9782
        179              4.5073   7.643146e-03         0.9761
        180              4.5073   7.462735e-03         0.9764
        181              4.5073   7.309771e-03         0.9795
        182              4.5073   7.223806e-03         0.9882
        183              4.5073   7.155744e-03         0.9906
        184              4.5073   7.058219e-03         0.9864
        185              4.5073   6.910895e-03         0.9791
        186              4.5073   6.705845e-03         0.9703
        187              4.5073   6.452930e-03         0.9623
        188              4.5073   6.324550e-03         0.9801
        189              4.5073   6.209451e-03         0.9818
        190              4.5073   6.117331e-03         0.9852
        191              4.5073   6.002762e-03         0.9813
        192              4.5073   5.795177e-03         0.9654
        193              4.5073   5.617591e-03         0.9694
        194              4.5073   5.492606e-03         0.9778
        195              4.5073   5.417069e-03         0.9862
        196              4.5073   5.365974e-03         0.9906
        197              4.5073   5.306792e-03         0.9890
        198              4.5073   5.199231e-03         0.9797
        199              4.5073   5.089314e-03         0.9789
        200              4.5073   4.984597e-03         0.9794
        201              4.5073   4.885234e-03         0.9801
        202              4.5073   4.831392e-03         0.9890
        203              4.5073   4.785620e-03         0.9905
        204              4.5073   4.720878e-03         0.9865
        205              4.5073   4.617083e-03         0.9780
        206              4.5073   4.473098e-03         0.9688
        207              4.5073   4.298896e-03         0.9611
        208              4.5073   4.211524e-03         0.9797
        209              4.5073   4.126273e-03         0.9798
        210              4.5073   4.054650e-03         0.9826
        211              4.5073   3.976860e-03         0.9808
        212              4.5073   3.832365e-03         0.9637
        213              4.5073   3.712143e-03         0.9686
        214              4.5073   3.627296e-03         0.9771
        215              4.5073   3.576847e-03         0.9861
        216              4.5073   3.542947e-03         0.9905
        217              4.5073   3.507197e-03         0.9899
        218              4.5073   3.438890e-03         0.9805
        219              4.5073   3.373210e-03         0.9809
        220              4.5073   3.309594e-03         0.9811
        221              4.5073   3.246149e-03         0.9808
        222              4.5073   3.212992e-03         0.9898
        223              4.5073   3.182623e-03         0.9905
        224              4.5073   3.138007e-03         0.9860
        225              4.5073   3.063484e-03         0.9763
        226              4.5073   2.962185e-03         0.9669
        227              4.5073   2.842631e-03         0.9596
        228              4.5073   2.782643e-03         0.9789
        229              4.5073   2.721328e-03         0.9780
        230              4.5073   2.669796e-03         0.9811
        231              4.5073   2.615614e-03         0.9797
        232              4.5073   2.516624e-03         0.9622
        233              4.5073   2.433075e-03         0.9668
        234              4.5073   2.372982e-03         0.9753
        235              4.5073   2.338936e-03         0.9857
        236              4.5073   2.316631e-03         0.9905
        237              4.5073   2.293694e-03         0.9901
        238              4.5073   2.248138e-03         0.9801
        239              4.5073   2.206995e-03         0.9817
        240              4.5073   2.167269e-03         0.9820
        241              4.5073   2.124825e-03         0.9804
        242              4.5073   2.103450e-03         0.9899
        243              4.5073   2.083674e-03         0.9906
        244              4.5073   2.054267e-03         0.9859
        245              4.5073   2.001011e-03         0.9741
        246              4.5073   1.930307e-03         0.9647
        247              4.5073   1.848721e-03         0.9577
        248              4.5073   1.808268e-03         0.9781
        249              4.5073   1.763726e-03         0.9754
        250              4.5073   1.726539e-03         0.9789
        251              4.5073   1.689947e-03         0.9788
        252              4.5073   1.624000e-03         0.9610
        253              4.5073   1.565009e-03         0.9637
        254              4.5073   1.522738e-03         0.9730
        255              4.5073   1.500731e-03         0.9855
        256              4.5073   1.486607e-03         0.9906
        257              4.5073   1.472336e-03         0.9904
        258              4.5073   1.442995e-03         0.9801
        259              4.5073   1.401880e-03         0.9715
        260              4.5073   1.363245e-03         0.9724
        261              4.5073   1.338839e-03         0.9821
        262              4.5073   1.324971e-03         0.9896
        263              4.5073   1.314374e-03         0.9920
        264              4.5073   1.297219e-03         0.9869
        265              4.5073   1.265595e-03         0.9756
        266              4.5073   1.219403e-03         0.9635
        267              4.5073   1.165827e-03         0.9561
        268              4.5073   1.136158e-03         0.9746
        269              4.5073   1.106047e-03         0.9735
        270              4.5073   1.081060e-03         0.9774
        271              4.5073   1.057849e-03         0.9785
        272              4.5073   1.011653e-03         0.9563
        273              4.5073   9.744879e-04         0.9633
        274              4.5073   9.499882e-04         0.9749
        275              4.5073   9.369585e-04         0.9863
        276              4.5073   9.290940e-04         0.9916
        277              4.5073   9.204583e-04         0.9907
        278              4.5073   9.048393e-04         0.9830
        279              4.5073   8.898536e-04         0.9834
        280              4.5073   8.755330e-04         0.9839
        281              4.5073   8.611554e-04         0.9836
        282              4.5073   8.530492e-04         0.9906
        283              4.5073   8.459542e-04         0.9917
        284              4.5073   8.339620e-04         0.9858
        285              4.5073   8.123608e-04         0.9741
        286              4.5073   7.823079e-04         0.9630
        287              4.5073   7.475207e-04         0.9555
        288              4.5073   7.289601e-04         0.9752
        289              4.5073   7.082735e-04         0.9716
        290              4.5073   6.906207e-04         0.9751
        291              4.5073   6.748595e-04         0.9772
        292              4.5073   6.470612e-04         0.9588
        293              4.5073   6.235908e-04         0.9637
        294              4.5073   6.062682e-04         0.9722
        295              4.5073   5.972097e-04         0.9851
        296              4.5073   5.916863e-04         0.9908
        297              4.5073   5.861331e-04         0.9906
        298              4.5073   5.771284e-04         0.9846
        299              4.5073   5.698246e-04         0.9873
        300              4.5073   5.629160e-04         0.9879
        301              4.5073   5.548604e-04         0.9857
        302              4.5073   5.496628e-04         0.9906
        303              4.5073   5.443566e-04         0.9903
        304              4.5073   5.351942e-04         0.9832
        305              4.5073   5.197481e-04         0.9711
        306              4.5073   5.013518e-04         0.9646
        307              4.5073   4.820704e-04         0.9615
        308              4.5073   4.703863e-04         0.9758
        309              4.5073   4.551120e-04         0.9675
        310              4.5073   4.420310e-04         0.9713
        311              4.5073   4.321963e-04         0.9778
        312              4.5073   4.173260e-04         0.9656
        313              4.5073   4.027740e-04         0.9651
        314              4.5073   3.902635e-04         0.9689
        315              4.5073   3.834007e-04         0.9824
        316              4.5073   3.794524e-04         0.9897
        317              4.5073   3.759048e-04         0.9907
        318              4.5073   3.706544e-04         0.9860
        319              4.5073   3.662881e-04         0.9882
        320              4.5073   3.621661e-04         0.9887
        321              4.5073   3.575503e-04         0.9873
        322              4.5073   3.542326e-04         0.9907
        323              4.5073   3.505049e-04         0.9895
        324              4.5073   3.438591e-04         0.9810
        325              4.5073   3.327386e-04         0.9677
        326              4.5073   3.209327e-04         0.9645
        327              4.5073   3.097232e-04         0.9651
        328              4.5073   3.025474e-04         0.9768
        329              4.5073   2.915968e-04         0.9638
        330              4.5073   2.822276e-04         0.9679
        331              4.5073   2.758958e-04         0.9776
        332              4.5073   2.670500e-04         0.9679
        333              4.5073   2.577056e-04         0.9650
        334              4.5073   2.490600e-04         0.9665
        335              4.5073   2.442267e-04         0.9806
        336              4.5073   2.416747e-04         0.9896
        337              4.5073   2.395729e-04         0.9913
        338              4.5073   2.366283e-04         0.9877
        339              4.5073   2.329755e-04         0.9846
        340              4.5073   2.295243e-04         0.9852
        341              4.5073   2.271336e-04         0.9896
        342              4.5073   2.249478e-04         0.9904
        343              4.5073   2.225149e-04         0.9892
        344              4.5073   2.183825e-04         0.9814
        345              4.5073   2.115974e-04         0.9689
        346              4.5073   2.037608e-04         0.9630
        347              4.5073   1.961096e-04         0.9624
        348              4.5073   1.915623e-04         0.9768
        349              4.5073   1.844891e-04         0.9631
        350              4.5073   1.789415e-04         0.9699
        351              4.5073   1.747135e-04         0.9764
        352              4.5073   1.685326e-04         0.9646
        353              4.5073   1.625619e-04         0.9646
        354              4.5073   1.573789e-04         0.9681
        355              4.5073   1.543450e-04         0.9807
        356              4.5073   1.527411e-04         0.9896
        357              4.5073   1.514816e-04         0.9918
        358              4.5073   1.498667e-04         0.9893
        359              4.5073   1.482822e-04         0.9894
        360              4.5073   1.467254e-04         0.9895
        361              4.5073   1.453226e-04         0.9904
        362              4.5073   1.440473e-04         0.9912
        363              4.5073   1.425693e-04         0.9897
        364              4.5073   1.399659e-04         0.9817
        365              4.5073   1.354891e-04         0.9680
        366              4.5073   1.304395e-04         0.9627
        367              4.5073   1.255801e-04         0.9627
        368              4.5073   1.224384e-04         0.9750
        369              4.5073   1.178715e-04         0.9627
        370              4.5073   1.141439e-04         0.9684
        371              4.5073   1.114363e-04         0.9763
        372              4.5073   1.074004e-04         0.9638
        373              4.5073   1.037559e-04         0.9661
        374              4.5073   1.005130e-04         0.9687
        375              4.5073   9.864468e-05         0.9814
        376              4.5073   9.765982e-05         0.9900
        377              4.5073   9.693787e-05         0.9926
        378              4.5073   9.603671e-05         0.9907
        379              4.5073   9.536281e-05         0.9930
        380              4.5073   9.470601e-05         0.9931
        381              4.5073   9.386835e-05         0.9912
        382              4.5073   9.312684e-05         0.9921
        383              4.5073   9.220860e-05         0.9901
        384              4.5073   9.051789e-05         0.9817
        385              4.5073   8.757986e-05         0.9675
        386              4.5073   8.441697e-05         0.9639
        387              4.5073   8.117176e-05         0.9616
        388              4.5073   7.917326e-05         0.9754
        389              4.5073   7.650414e-05         0.9663
        390              4.5073   7.415814e-05         0.9693
        391              4.5073   7.243978e-05         0.9768
        392              4.5073   6.992122e-05         0.9652
        393              4.5073   6.755464e-05         0.9662
        394              4.5073   6.550298e-05         0.9696
        395              4.5073   6.442420e-05         0.9835
        396              4.5073   6.386114e-05         0.9913
        397              4.5073   6.343917e-05         0.9934
        398              4.5073   6.291857e-05         0.9918
        399              4.5073   6.255101e-05         0.9942
        400              4.5073   6.218764e-05         0.9942
        401              4.5073   6.169533e-05         0.9921
        402              4.5073   6.126233e-05         0.9930
        403              4.5073   6.074052e-05         0.9915
        404              4.5073   5.976100e-05         0.9839
        405              4.5073   5.783132e-05         0.9677
        406              4.5073   5.582332e-05         0.9653
        407              4.5073   5.381541e-05         0.9640
        408              4.5073   5.256528e-05         0.9768
        409              4.5073   5.088652e-05         0.9681
        410              4.5073   4.942387e-05         0.9713
        411              4.5073   4.836018e-05         0.9785
        412              4.5073   4.681841e-05         0.9681
        413              4.5073   4.530922e-05         0.9678
        414              4.5073   4.403902e-05         0.9720
        415              4.5073   4.342640e-05         0.9861
        416              4.5073   4.311669e-05         0.9929
        417              4.5073   4.288333e-05         0.9946
        418              4.5073   4.258149e-05         0.9930
        419              4.5073   4.234146e-05         0.9944
        420              4.5073   4.210422e-05         0.9944
        421              4.5073   4.182142e-05         0.9933
        422              4.5073   4.157963e-05         0.9942
        423              4.5073   4.129297e-05         0.9931
        424              4.5073   4.075008e-05         0.9869
        425              4.5073   3.955404e-05         0.9706
        426              4.5073   3.826284e-05         0.9674
        427              4.5073   3.703658e-05         0.9680
        428              4.5073   3.627819e-05         0.9795
        429              4.5073   3.518842e-05         0.9700
        430              4.5073   3.427321e-05         0.9740
        431              4.5073   3.360933e-05         0.9806
        432              4.5073   3.266358e-05         0.9719
        433              4.5073   3.170647e-05         0.9707
        434              4.5073   3.092739e-05         0.9754
        435              4.5073   3.057961e-05         0.9888
        436              4.5073   3.040358e-05         0.9942
        437              4.5073   3.027417e-05         0.9957
        438              4.5073   3.010033e-05         0.9943
        439              4.5073   2.991853e-05         0.9940
        440              4.5073   2.973998e-05         0.9940
        441              4.5073   2.958330e-05         0.9947
        442              4.5073   2.944665e-05         0.9954
        443              4.5073   2.928091e-05         0.9944
        444              4.5073   2.897940e-05         0.9897
        445              4.5073   2.826589e-05         0.9754
        446              4.5073   2.743132e-05         0.9705
        447              4.5073   2.667133e-05         0.9723
        448              4.5073   2.615545e-05         0.9807
        449              4.5073   2.540920e-05         0.9715
        450              4.5073   2.485503e-05         0.9782
        451              4.5073   2.436528e-05         0.9803
        452              4.5073   2.373976e-05         0.9743
        453              4.5073   2.310008e-05         0.9731
        454              4.5073   2.261866e-05         0.9792
        455              4.5073   2.240689e-05         0.9906
        456              4.5073   2.229318e-05         0.9949
        457              4.5073   2.221321e-05         0.9964
        458              4.5073   2.210959e-05         0.9953
        459              4.5073   2.200217e-05         0.9951
        460              4.5073   2.189792e-05         0.9953
        461              4.5073   2.180671e-05         0.9958
        462              4.5073   2.171675e-05         0.9959
        463              4.5073   2.161412e-05         0.9953
        464              4.5073   2.143114e-05         0.9915
        465              4.5073   2.097284e-05         0.9786
        466              4.5073   2.038027e-05         0.9717
        467              4.5073   1.986920e-05         0.9749
        468              4.5073   1.948462e-05         0.9806
        469              4.5073   1.898508e-05         0.9744
        470              4.5073   1.857710e-05         0.9785
        471              4.5073   1.821628e-05         0.9806
        472              4.5073   1.782268e-05         0.9784
        473              4.5073   1.736033e-05         0.9741
        474              4.5073   1.698073e-05         0.9781
        475              4.5073   1.684377e-05         0.9919
        476              4.5073   1.676710e-05         0.9954
        477              4.5073   1.670770e-05         0.9965
        478              4.5073   1.663904e-05         0.9959
        479              4.5073   1.657661e-05         0.9962
        480              4.5073   1.651758e-05         0.9964
        481              4.5073   1.646211e-05         0.9966
        482              4.5073   1.639271e-05         0.9958
        483              4.5073   1.633027e-05         0.9962
        484              4.5073   1.622154e-05         0.9933
        485              4.5073   1.587925e-05         0.9789
        486              4.5073   1.541874e-05         0.9710
        487              4.5073   1.509442e-05         0.9790
        488              4.5073   1.477210e-05         0.9786
        489              4.5073   1.440651e-05         0.9753
        490              4.5073   1.411757e-05         0.9799
        491              4.5073   1.379559e-05         0.9772
        492              4.5073   1.354723e-05         0.9820
        493              4.5073   1.314833e-05         0.9706
     ----------------------------------------------------------------------
     Total Iterations: 494
     Avg Convergence Rate:               0.9724
     Final Residual:           1.314833e-05
     Total Reduction in Residual:      9.711549e-07
     Maximum Memory Usage:                4.507 GB
     ----------------------------------------------------------------------

Total Time: 16.6462 setup: 0.555882 s solve: 16.0903 s solve(per iteration): 0.0325714 s

_Output for PCG_AGGREGATION_JACOBI.json (~16s as well):_ AMGX version 2.5.0 Built on Apr 25 2024, 13:20:11 Compiled with CUDA Runtime 12.4, using CUDA driver 12.4 AMG Grid: Number of Levels: 9 LVL ROWS NNZ PARTS SPRSTY Mem (GB)

       0(D)      1507876         113305856      1  4.98e-05            1.3
       1(D)       281692          31103364      1  0.000392          0.706
       2(D)        39859           4883517      1   0.00307          0.111
       3(D)         4793            328295      1    0.0143        0.00753
       4(D)          762             35364      1    0.0609       0.000822
       5(D)          198              7160      1     0.183       0.000168
       6(D)           60              1622      1     0.451       3.88e-05
       7(D)           18               254      1     0.784       6.45e-06
       8(D)            6                36      1         1       9.98e-07
     ----------------------------------------------------------------------
     Grid Complexity: 1.21712
     Operator Complexity: 1.3209
     Total Memory Usage: 2.12977 GB
     ----------------------------------------------------------------------
       iter      Mem Usage (GB)       residual           rate
     ----------------------------------------------------------------------
        Ini             4.50732   1.353886e+01
          0             4.50732   9.552636e+00         0.7056
          1              4.5073   6.826744e+00         0.7146
          2              4.5073   5.451789e+00         0.7986
          3              4.5073   4.193551e+00         0.7692
          4              4.5073   3.318682e+00         0.7914
          5              4.5073   2.696656e+00         0.8126
          6              4.5073   2.253779e+00         0.8358
          7              4.5073   1.897404e+00         0.8419
          8              4.5073   1.605763e+00         0.8463
          9              4.5073   1.371794e+00         0.8543
         10              4.5073   1.280317e+00         0.9333
         11              4.5073   1.165342e+00         0.9102
         12              4.5073   1.044937e+00         0.8967
         13              4.5073   9.311771e-01         0.8911
         14              4.5073   8.317691e-01         0.8932
         15              4.5073   7.104873e-01         0.8542
         16              4.5073   6.342603e-01         0.8927
         17              4.5073   5.645147e-01         0.8900
         18              4.5073   5.062215e-01         0.8967
         19              4.5073   4.567174e-01         0.9022
         20              4.5073   4.388230e-01         0.9608
         21              4.5073   4.211402e-01         0.9597
         22              4.5073   3.973845e-01         0.9436
         23              4.5073   3.710982e-01         0.9339
         24              4.5073   3.424965e-01         0.9229
         25              4.5073   3.155958e-01         0.9215
         26              4.5073   2.966112e-01         0.9398
         27              4.5073   2.795076e-01         0.9423
         28              4.5073   2.635899e-01         0.9431
         29              4.5073   2.461252e-01         0.9337
         30              4.5073   2.365946e-01         0.9613
         31              4.5073   2.264181e-01         0.9570
         32              4.5073   2.144454e-01         0.9471
         33              4.5073   2.034822e-01         0.9489
         34              4.5073   1.939666e-01         0.9532
         35              4.5073   1.830933e-01         0.9439
         36              4.5073   1.759580e-01         0.9610
         37              4.5073   1.689407e-01         0.9601
         38              4.5073   1.625273e-01         0.9620
         39              4.5073   1.567911e-01         0.9647
         40              4.5073   1.523330e-01         0.9716
         41              4.5073   1.482169e-01         0.9730
         42              4.5073   1.430960e-01         0.9655
         43              4.5073   1.379123e-01         0.9638
         44              4.5073   1.318838e-01         0.9563
         45              4.5073   1.262892e-01         0.9576
         46              4.5073   1.216729e-01         0.9634
         47              4.5073   1.172642e-01         0.9638
         48              4.5073   1.136271e-01         0.9690
         49              4.5073   1.098468e-01         0.9667
         50              4.5073   1.073779e-01         0.9775
         51              4.5073   1.044496e-01         0.9727
         52              4.5073   1.007739e-01         0.9648
         53              4.5073   9.737754e-02         0.9663
         54              4.5073   9.470861e-02         0.9726
         55              4.5073   9.166303e-02         0.9678
         56              4.5073   8.969872e-02         0.9786
         57              4.5073   8.761253e-02         0.9767
         58              4.5073   8.539046e-02         0.9746
         59              4.5073   8.346770e-02         0.9775
         60              4.5073   8.175577e-02         0.9795
         61              4.5073   8.011439e-02         0.9799
         62              4.5073   7.851187e-02         0.9800
         63              4.5073   7.699035e-02         0.9806
         64              4.5073   7.499845e-02         0.9741
         65              4.5073   7.315360e-02         0.9754
         66              4.5073   7.119191e-02         0.9732
         67              4.5073   6.927591e-02         0.9731
         68              4.5073   6.783679e-02         0.9792
         69              4.5073   6.647737e-02         0.9800
         70              4.5073   6.545250e-02         0.9846
         71              4.5073   6.426370e-02         0.9818
         72              4.5073   6.257041e-02         0.9737
         73              4.5073   6.100275e-02         0.9749
         74              4.5073   5.987852e-02         0.9816
         75              4.5073   5.862508e-02         0.9791
         76              4.5073   5.780494e-02         0.9860
         77              4.5073   5.687329e-02         0.9839
         78              4.5073   5.563071e-02         0.9782
         79              4.5073   5.456461e-02         0.9808
         80              4.5073   5.351356e-02         0.9807
         81              4.5073   5.255514e-02         0.9821
         82              4.5073   5.177287e-02         0.9851
         83              4.5073   5.106999e-02         0.9864
         84              4.5073   5.012493e-02         0.9815
         85              4.5073   4.921204e-02         0.9818
         86              4.5073   4.799043e-02         0.9752
         87              4.5073   4.670630e-02         0.9732
         88              4.5073   4.583710e-02         0.9814
         89              4.5073   4.502878e-02         0.9824
         90              4.5073   4.445102e-02         0.9872
         91              4.5073   4.367765e-02         0.9826
         92              4.5073   4.254507e-02         0.9741
         93              4.5073   4.147386e-02         0.9748
         94              4.5073   4.078047e-02         0.9833
         95              4.5073   4.004918e-02         0.9821
         96              4.5073   3.958986e-02         0.9885
         97              4.5073   3.904670e-02         0.9863
         98              4.5073   3.819285e-02         0.9781
         99              4.5073   3.743150e-02         0.9801
        100              4.5073   3.671366e-02         0.9808
        101              4.5073   3.597912e-02         0.9800
        102              4.5073   3.552583e-02         0.9874
        103              4.5073   3.510992e-02         0.9883
        104              4.5073   3.452025e-02         0.9832
        105              4.5073   3.394837e-02         0.9834
        106              4.5073   3.301806e-02         0.9726
        107              4.5073   3.206440e-02         0.9711
        108              4.5073   3.147588e-02         0.9816
        109              4.5073   3.095063e-02         0.9833
        110              4.5073   3.051699e-02         0.9860
        111              4.5073   3.003388e-02         0.9842
        112              4.5073   2.918968e-02         0.9719
        113              4.5073   2.837800e-02         0.9722
        114              4.5073   2.792080e-02         0.9839
        115              4.5073   2.745213e-02         0.9832
        116              4.5073   2.715536e-02         0.9892
        117              4.5073   2.684906e-02         0.9887
        118              4.5073   2.618969e-02         0.9754
        119              4.5073   2.561850e-02         0.9782
        120              4.5073   2.506056e-02         0.9782
        121              4.5073   2.451296e-02         0.9781
        122              4.5073   2.423323e-02         0.9886
        123              4.5073   2.397209e-02         0.9892
        124              4.5073   2.358096e-02         0.9837
        125              4.5073   2.317335e-02         0.9827
        126              4.5073   2.252357e-02         0.9720
        127              4.5073   2.182169e-02         0.9688
        128              4.5073   2.142175e-02         0.9817
        129              4.5073   2.106657e-02         0.9834
        130              4.5073   2.076983e-02         0.9859
        131              4.5073   2.043719e-02         0.9840
        132              4.5073   1.982875e-02         0.9702
        133              4.5073   1.925887e-02         0.9713
        134              4.5073   1.892543e-02         0.9827
        135              4.5073   1.861500e-02         0.9836
        136              4.5073   1.841600e-02         0.9893
        137              4.5073   1.822721e-02         0.9897
        138              4.5073   1.774987e-02         0.9738
        139              4.5073   1.723753e-02         0.9711
        140              4.5073   1.675373e-02         0.9719
        141              4.5073   1.636719e-02         0.9769
        142              4.5073   1.617550e-02         0.9883
        143              4.5073   1.600921e-02         0.9897
        144              4.5073   1.578544e-02         0.9860
        145              4.5073   1.549496e-02         0.9816
        146              4.5073   1.508306e-02         0.9734
        147              4.5073   1.456481e-02         0.9656
        148              4.5073   1.427043e-02         0.9798
        149              4.5073   1.402647e-02         0.9829
        150              4.5073   1.384524e-02         0.9871
        151              4.5073   1.359538e-02         0.9820
        152              4.5073   1.315202e-02         0.9674
        153              4.5073   1.278672e-02         0.9722
        154              4.5073   1.253887e-02         0.9806
        155              4.5073   1.235779e-02         0.9856
        156              4.5073   1.223345e-02         0.9899
        157              4.5073   1.210488e-02         0.9895
        158              4.5073   1.180894e-02         0.9756
        159              4.5073   1.147988e-02         0.9721
        160              4.5073   1.116957e-02         0.9730
        161              4.5073   1.092455e-02         0.9781
        162              4.5073   1.079219e-02         0.9879
        163              4.5073   1.069048e-02         0.9906
        164              4.5073   1.054792e-02         0.9867
        165              4.5073   1.033906e-02         0.9802
        166              4.5073   1.006528e-02         0.9735
        167              4.5073   9.688645e-03         0.9626
        168              4.5073   9.485966e-03         0.9791
        169              4.5073   9.321227e-03         0.9826
        170              4.5073   9.195419e-03         0.9865
        171              4.5073   9.021929e-03         0.9811
        172              4.5073   8.718005e-03         0.9663
        173              4.5073   8.466200e-03         0.9711
        174              4.5073   8.285718e-03         0.9787
        175              4.5073   8.173952e-03         0.9865
        176              4.5073   8.095984e-03         0.9905
        177              4.5073   8.004101e-03         0.9887
        178              4.5073   7.829951e-03         0.9782
        179              4.5073   7.643146e-03         0.9761
        180              4.5073   7.462735e-03         0.9764
        181              4.5073   7.309771e-03         0.9795
        182              4.5073   7.223806e-03         0.9882
        183              4.5073   7.155744e-03         0.9906
        184              4.5073   7.058219e-03         0.9864
        185              4.5073   6.910895e-03         0.9791
        186              4.5073   6.705845e-03         0.9703
        187              4.5073   6.452930e-03         0.9623
        188              4.5073   6.324550e-03         0.9801
        189              4.5073   6.209451e-03         0.9818
        190              4.5073   6.117331e-03         0.9852
        191              4.5073   6.002762e-03         0.9813
        192              4.5073   5.795177e-03         0.9654
        193              4.5073   5.617591e-03         0.9694
        194              4.5073   5.492606e-03         0.9778
        195              4.5073   5.417069e-03         0.9862
        196              4.5073   5.365974e-03         0.9906
        197              4.5073   5.306792e-03         0.9890
        198              4.5073   5.199231e-03         0.9797
        199              4.5073   5.089314e-03         0.9789
        200              4.5073   4.984597e-03         0.9794
        201              4.5073   4.885234e-03         0.9801
        202              4.5073   4.831392e-03         0.9890
        203              4.5073   4.785620e-03         0.9905
        204              4.5073   4.720878e-03         0.9865
        205              4.5073   4.617083e-03         0.9780
        206              4.5073   4.473098e-03         0.9688
        207              4.5073   4.298896e-03         0.9611
        208              4.5073   4.211524e-03         0.9797
        209              4.5073   4.126273e-03         0.9798
        210              4.5073   4.054650e-03         0.9826
        211              4.5073   3.976860e-03         0.9808
        212              4.5073   3.832365e-03         0.9637
        213              4.5073   3.712143e-03         0.9686
        214              4.5073   3.627296e-03         0.9771
        215              4.5073   3.576847e-03         0.9861
        216              4.5073   3.542947e-03         0.9905
        217              4.5073   3.507197e-03         0.9899
        218              4.5073   3.438890e-03         0.9805
        219              4.5073   3.373210e-03         0.9809
        220              4.5073   3.309594e-03         0.9811
        221              4.5073   3.246149e-03         0.9808
        222              4.5073   3.212992e-03         0.9898
        223              4.5073   3.182623e-03         0.9905
        224              4.5073   3.138007e-03         0.9860
        225              4.5073   3.063484e-03         0.9763
        226              4.5073   2.962185e-03         0.9669
        227              4.5073   2.842631e-03         0.9596
        228              4.5073   2.782643e-03         0.9789
        229              4.5073   2.721328e-03         0.9780
        230              4.5073   2.669796e-03         0.9811
        231              4.5073   2.615614e-03         0.9797
        232              4.5073   2.516624e-03         0.9622
        233              4.5073   2.433075e-03         0.9668
        234              4.5073   2.372982e-03         0.9753
        235              4.5073   2.338936e-03         0.9857
        236              4.5073   2.316631e-03         0.9905
        237              4.5073   2.293694e-03         0.9901
        238              4.5073   2.248138e-03         0.9801
        239              4.5073   2.206995e-03         0.9817
        240              4.5073   2.167269e-03         0.9820
        241              4.5073   2.124825e-03         0.9804
        242              4.5073   2.103450e-03         0.9899
        243              4.5073   2.083674e-03         0.9906
        244              4.5073   2.054267e-03         0.9859
        245              4.5073   2.001011e-03         0.9741
        246              4.5073   1.930307e-03         0.9647
        247              4.5073   1.848721e-03         0.9577
        248              4.5073   1.808268e-03         0.9781
        249              4.5073   1.763726e-03         0.9754
        250              4.5073   1.726539e-03         0.9789
        251              4.5073   1.689947e-03         0.9788
        252              4.5073   1.624000e-03         0.9610
        253              4.5073   1.565009e-03         0.9637
        254              4.5073   1.522738e-03         0.9730
        255              4.5073   1.500731e-03         0.9855
        256              4.5073   1.486607e-03         0.9906
        257              4.5073   1.472336e-03         0.9904
        258              4.5073   1.442995e-03         0.9801
        259              4.5073   1.401880e-03         0.9715
        260              4.5073   1.363245e-03         0.9724
        261              4.5073   1.338839e-03         0.9821
        262              4.5073   1.324971e-03         0.9896
        263              4.5073   1.314374e-03         0.9920
        264              4.5073   1.297219e-03         0.9869
        265              4.5073   1.265595e-03         0.9756
        266              4.5073   1.219403e-03         0.9635
        267              4.5073   1.165827e-03         0.9561
        268              4.5073   1.136158e-03         0.9746
        269              4.5073   1.106047e-03         0.9735
        270              4.5073   1.081060e-03         0.9774
        271              4.5073   1.057849e-03         0.9785
        272              4.5073   1.011653e-03         0.9563
        273              4.5073   9.744879e-04         0.9633
        274              4.5073   9.499882e-04         0.9749
        275              4.5073   9.369585e-04         0.9863
        276              4.5073   9.290940e-04         0.9916
        277              4.5073   9.204583e-04         0.9907
        278              4.5073   9.048393e-04         0.9830
        279              4.5073   8.898536e-04         0.9834
        280              4.5073   8.755330e-04         0.9839
        281              4.5073   8.611554e-04         0.9836
        282              4.5073   8.530492e-04         0.9906
        283              4.5073   8.459542e-04         0.9917
        284              4.5073   8.339620e-04         0.9858
        285              4.5073   8.123608e-04         0.9741
        286              4.5073   7.823079e-04         0.9630
        287              4.5073   7.475207e-04         0.9555
        288              4.5073   7.289601e-04         0.9752
        289              4.5073   7.082735e-04         0.9716
        290              4.5073   6.906207e-04         0.9751
        291              4.5073   6.748595e-04         0.9772
        292              4.5073   6.470612e-04         0.9588
        293              4.5073   6.235908e-04         0.9637
        294              4.5073   6.062682e-04         0.9722
        295              4.5073   5.972097e-04         0.9851
        296              4.5073   5.916863e-04         0.9908
        297              4.5073   5.861331e-04         0.9906
        298              4.5073   5.771284e-04         0.9846
        299              4.5073   5.698246e-04         0.9873
        300              4.5073   5.629160e-04         0.9879
        301              4.5073   5.548604e-04         0.9857
        302              4.5073   5.496628e-04         0.9906
        303              4.5073   5.443566e-04         0.9903
        304              4.5073   5.351942e-04         0.9832
        305              4.5073   5.197481e-04         0.9711
        306              4.5073   5.013518e-04         0.9646
        307              4.5073   4.820704e-04         0.9615
        308              4.5073   4.703863e-04         0.9758
        309              4.5073   4.551120e-04         0.9675
        310              4.5073   4.420310e-04         0.9713
        311              4.5073   4.321963e-04         0.9778
        312              4.5073   4.173260e-04         0.9656
        313              4.5073   4.027740e-04         0.9651
        314              4.5073   3.902635e-04         0.9689
        315              4.5073   3.834007e-04         0.9824
        316              4.5073   3.794524e-04         0.9897
        317              4.5073   3.759048e-04         0.9907
        318              4.5073   3.706544e-04         0.9860
        319              4.5073   3.662881e-04         0.9882
        320              4.5073   3.621661e-04         0.9887
        321              4.5073   3.575503e-04         0.9873
        322              4.5073   3.542326e-04         0.9907
        323              4.5073   3.505049e-04         0.9895
        324              4.5073   3.438591e-04         0.9810
        325              4.5073   3.327386e-04         0.9677
        326              4.5073   3.209327e-04         0.9645
        327              4.5073   3.097232e-04         0.9651
        328              4.5073   3.025474e-04         0.9768
        329              4.5073   2.915968e-04         0.9638
        330              4.5073   2.822276e-04         0.9679
        331              4.5073   2.758958e-04         0.9776
        332              4.5073   2.670500e-04         0.9679
        333              4.5073   2.577056e-04         0.9650
        334              4.5073   2.490600e-04         0.9665
        335              4.5073   2.442267e-04         0.9806
        336              4.5073   2.416747e-04         0.9896
        337              4.5073   2.395729e-04         0.9913
        338              4.5073   2.366283e-04         0.9877
        339              4.5073   2.329755e-04         0.9846
        340              4.5073   2.295243e-04         0.9852
        341              4.5073   2.271336e-04         0.9896
        342              4.5073   2.249478e-04         0.9904
        343              4.5073   2.225149e-04         0.9892
        344              4.5073   2.183825e-04         0.9814
        345              4.5073   2.115974e-04         0.9689
        346              4.5073   2.037608e-04         0.9630
        347              4.5073   1.961096e-04         0.9624
        348              4.5073   1.915623e-04         0.9768
        349              4.5073   1.844891e-04         0.9631
        350              4.5073   1.789415e-04         0.9699
        351              4.5073   1.747135e-04         0.9764
        352              4.5073   1.685326e-04         0.9646
        353              4.5073   1.625619e-04         0.9646
        354              4.5073   1.573789e-04         0.9681
        355              4.5073   1.543450e-04         0.9807
        356              4.5073   1.527411e-04         0.9896
        357              4.5073   1.514816e-04         0.9918
        358              4.5073   1.498667e-04         0.9893
        359              4.5073   1.482822e-04         0.9894
        360              4.5073   1.467254e-04         0.9895
        361              4.5073   1.453226e-04         0.9904
        362              4.5073   1.440473e-04         0.9912
        363              4.5073   1.425693e-04         0.9897
        364              4.5073   1.399659e-04         0.9817
        365              4.5073   1.354891e-04         0.9680
        366              4.5073   1.304395e-04         0.9627
        367              4.5073   1.255801e-04         0.9627
        368              4.5073   1.224384e-04         0.9750
        369              4.5073   1.178715e-04         0.9627
        370              4.5073   1.141439e-04         0.9684
        371              4.5073   1.114363e-04         0.9763
        372              4.5073   1.074004e-04         0.9638
        373              4.5073   1.037559e-04         0.9661
        374              4.5073   1.005130e-04         0.9687
        375              4.5073   9.864467e-05         0.9814
        376              4.5073   9.765982e-05         0.9900
        377              4.5073   9.693787e-05         0.9926
        378              4.5073   9.603671e-05         0.9907
        379              4.5073   9.536281e-05         0.9930
        380              4.5073   9.470601e-05         0.9931
        381              4.5073   9.386835e-05         0.9912
        382              4.5073   9.312684e-05         0.9921
        383              4.5073   9.220860e-05         0.9901
        384              4.5073   9.051789e-05         0.9817
        385              4.5073   8.757986e-05         0.9675
        386              4.5073   8.441697e-05         0.9639
        387              4.5073   8.117176e-05         0.9616
        388              4.5073   7.917326e-05         0.9754
        389              4.5073   7.650414e-05         0.9663
        390              4.5073   7.415814e-05         0.9693
        391              4.5073   7.243978e-05         0.9768
        392              4.5073   6.992122e-05         0.9652
        393              4.5073   6.755464e-05         0.9662
        394              4.5073   6.550298e-05         0.9696
        395              4.5073   6.442420e-05         0.9835
        396              4.5073   6.386114e-05         0.9913
        397              4.5073   6.343917e-05         0.9934
        398              4.5073   6.291857e-05         0.9918
        399              4.5073   6.255101e-05         0.9942
        400              4.5073   6.218764e-05         0.9942
        401              4.5073   6.169533e-05         0.9921
        402              4.5073   6.126233e-05         0.9930
        403              4.5073   6.074052e-05         0.9915
        404              4.5073   5.976100e-05         0.9839
        405              4.5073   5.783132e-05         0.9677
        406              4.5073   5.582332e-05         0.9653
        407              4.5073   5.381541e-05         0.9640
        408              4.5073   5.256528e-05         0.9768
        409              4.5073   5.088652e-05         0.9681
        410              4.5073   4.942387e-05         0.9713
        411              4.5073   4.836018e-05         0.9785
        412              4.5073   4.681841e-05         0.9681
        413              4.5073   4.530922e-05         0.9678
        414              4.5073   4.403902e-05         0.9720
        415              4.5073   4.342640e-05         0.9861
        416              4.5073   4.311669e-05         0.9929
        417              4.5073   4.288333e-05         0.9946
        418              4.5073   4.258149e-05         0.9930
        419              4.5073   4.234146e-05         0.9944
        420              4.5073   4.210422e-05         0.9944
        421              4.5073   4.182142e-05         0.9933
        422              4.5073   4.157963e-05         0.9942
        423              4.5073   4.129297e-05         0.9931
        424              4.5073   4.075008e-05         0.9869
        425              4.5073   3.955404e-05         0.9706
        426              4.5073   3.826284e-05         0.9674
        427              4.5073   3.703658e-05         0.9680
        428              4.5073   3.627819e-05         0.9795
        429              4.5073   3.518842e-05         0.9700
        430              4.5073   3.427321e-05         0.9740
        431              4.5073   3.360933e-05         0.9806
        432              4.5073   3.266358e-05         0.9719
        433              4.5073   3.170647e-05         0.9707
        434              4.5073   3.092739e-05         0.9754
        435              4.5073   3.057961e-05         0.9888
        436              4.5073   3.040358e-05         0.9942
        437              4.5073   3.027417e-05         0.9957
        438              4.5073   3.010033e-05         0.9943
        439              4.5073   2.991853e-05         0.9940
        440              4.5073   2.973998e-05         0.9940
        441              4.5073   2.958330e-05         0.9947
        442              4.5073   2.944665e-05         0.9954
        443              4.5073   2.928091e-05         0.9944
        444              4.5073   2.897939e-05         0.9897
        445              4.5073   2.826589e-05         0.9754
        446              4.5073   2.743131e-05         0.9705
        447              4.5073   2.667133e-05         0.9723
        448              4.5073   2.615545e-05         0.9807
        449              4.5073   2.540920e-05         0.9715
        450              4.5073   2.485503e-05         0.9782
        451              4.5073   2.436528e-05         0.9803
        452              4.5073   2.373976e-05         0.9743
        453              4.5073   2.310008e-05         0.9731
        454              4.5073   2.261866e-05         0.9792
        455              4.5073   2.240689e-05         0.9906
        456              4.5073   2.229318e-05         0.9949
        457              4.5073   2.221321e-05         0.9964
        458              4.5073   2.210959e-05         0.9953
        459              4.5073   2.200217e-05         0.9951
        460              4.5073   2.189792e-05         0.9953
        461              4.5073   2.180671e-05         0.9958
        462              4.5073   2.171675e-05         0.9959
        463              4.5073   2.161412e-05         0.9953
        464              4.5073   2.143114e-05         0.9915
        465              4.5073   2.097284e-05         0.9786
        466              4.5073   2.038027e-05         0.9717
        467              4.5073   1.986920e-05         0.9749
        468              4.5073   1.948462e-05         0.9806
        469              4.5073   1.898509e-05         0.9744
        470              4.5073   1.857711e-05         0.9785
        471              4.5073   1.821628e-05         0.9806
        472              4.5073   1.782268e-05         0.9784
        473              4.5073   1.736033e-05         0.9741
        474              4.5073   1.698073e-05         0.9781
        475              4.5073   1.684377e-05         0.9919
        476              4.5073   1.676710e-05         0.9954
        477              4.5073   1.670770e-05         0.9965
        478              4.5073   1.663904e-05         0.9959
        479              4.5073   1.657661e-05         0.9962
        480              4.5073   1.651758e-05         0.9964
        481              4.5073   1.646211e-05         0.9966
        482              4.5073   1.639271e-05         0.9958
        483              4.5073   1.633027e-05         0.9962
        484              4.5073   1.622154e-05         0.9933
        485              4.5073   1.587925e-05         0.9789
        486              4.5073   1.541874e-05         0.9710
        487              4.5073   1.509443e-05         0.9790
        488              4.5073   1.477211e-05         0.9786
        489              4.5073   1.440650e-05         0.9753
        490              4.5073   1.411756e-05         0.9799
        491              4.5073   1.379558e-05         0.9772
        492              4.5073   1.354723e-05         0.9820
        493              4.5073   1.314832e-05         0.9706
     ----------------------------------------------------------------------
     Total Iterations: 494
     Avg Convergence Rate:               0.9724
     Final Residual:           1.314832e-05
     Total Reduction in Residual:      9.711542e-07
     Maximum Memory Usage:                4.507 GB
     ----------------------------------------------------------------------

Total Time: 16.4863 setup: 0.550713 s solve: 15.9356 s solve(per iteration): 0.0322583 s

_Adding strength threshold = 0.6 i manage to get GMRES_AMD_D2 to get to 7s but still longer than CG without preconditioner._ { "config_version": 2, "determinism_flag": 1, "exception_handling" : 1, "solver": { "print_grid_stats": 1, "store_res_history": 0, "solver": "GMRES", "print_solve_stats": 1, "obtain_timings": 1, "preconditioner": { "interpolator": "D2", "print_grid_stats": 1, "solver": "AMG", "smoother": "JACOBI_L1", "presweeps": 2, "selector": "PMIS", "coarsest_sweeps": 2, "coarse_solver": "NOSOLVER", "max_iters": 1, "interp_max_elements": 4, "min_coarse_rows": 2, "scope": "amg_solver", "max_levels": 24, "cycle": "V", "postsweeps": 2, "max_row_sum": 0.9, "strength_threshold": 0.6 }, "max_iters": 1000, "monitor_residual": 1, "gmres_n_restart": 10, "convergence": "RELATIVE_INI_CORE", "tolerance": 1e-06, "norm": "L2" } } Here is the out put: AMGX version 2.5.0 Built on Apr 25 2024, 13:20:11 Compiled with CUDA Runtime 12.4, using CUDA driver 12.4 AMG Grid: Number of Levels: 8 LVL ROWS NNZ PARTS SPRSTY Mem (GB)

       0(D)      1507876         113305856      1  4.98e-05           1.31
       1(D)       427806          45533312      1  0.000249           1.04
       2(D)       137879          22690729      1   0.00119          0.513
       3(D)        38257           6494695      1   0.00444          0.147
       4(D)         9256           1114344      1     0.013         0.0253
       5(D)         1820            124304      1    0.0375        0.00285
       6(D)          351             12935      1     0.105       0.000303
       7(D)           47               451      1     0.204       1.15e-05
     ----------------------------------------------------------------------
     Grid Complexity: 1.40813
     Operator Complexity: 1.67049
     Total Memory Usage: 3.03017 GB
     ----------------------------------------------------------------------
       iter      Mem Usage (GB)       residual           rate
     ----------------------------------------------------------------------
        Ini             4.52881   1.353886e+01
          0             4.52881   8.768310e+00         0.6476
          1              4.5288   5.875893e+00         0.6701
          2              4.5288   4.114999e+00         0.7003
          3              4.5288   2.921429e+00         0.7099
          4              4.5288   2.161990e+00         0.7400
          5              4.5288   1.646122e+00         0.7614
          6              4.5288   1.277718e+00         0.7762
          7              4.5288   1.001020e+00         0.7834
          8              4.5288   7.980492e-01         0.7972
          9              4.5288   6.559208e-01         0.8219
         10              4.5288   5.960254e-01         0.9087
         11              4.5288   5.332775e-01         0.8947
         12              4.5288   4.738347e-01         0.8885
         13              4.5288   4.126436e-01         0.8709
         14              4.5288   3.498271e-01         0.8478
         15              4.5288   2.893011e-01         0.8270
         16              4.5288   2.489482e-01         0.8605
         17              4.5288   2.146483e-01         0.8622
         18              4.5288   1.837306e-01         0.8560
         19              4.5288   1.614325e-01         0.8786
         20              4.5288   1.520154e-01         0.9417
         21              4.5288   1.407548e-01         0.9259
         22              4.5288   1.286014e-01         0.9137
         23              4.5288   1.176749e-01         0.9150
         24              4.5288   1.072537e-01         0.9114
         25              4.5288   9.944967e-02         0.9272
         26              4.5288   9.291163e-02         0.9343
         27              4.5288   8.549769e-02         0.9202
         28              4.5288   7.896385e-02         0.9236
         29              4.5288   7.237665e-02         0.9166
         30              4.5288   6.818068e-02         0.9420
         31              4.5288   6.386154e-02         0.9367
         32              4.5288   5.982216e-02         0.9367
         33              4.5288   5.638698e-02         0.9426
         34              4.5288   5.283324e-02         0.9370
         35              4.5288   4.850473e-02         0.9181
         36              4.5288   4.445382e-02         0.9165
         37              4.5288   4.021743e-02         0.9047
         38              4.5288   3.686018e-02         0.9165
         39              4.5288   3.463181e-02         0.9395
         40              4.5288   3.264420e-02         0.9426
         41              4.5288   3.038673e-02         0.9308
         42              4.5288   2.781750e-02         0.9154
         43              4.5288   2.567715e-02         0.9231
         44              4.5288   2.368404e-02         0.9224
         45              4.5288   2.225356e-02         0.9396
         46              4.5288   2.094205e-02         0.9411
         47              4.5288   1.957047e-02         0.9345
         48              4.5288   1.835569e-02         0.9379
         49              4.5288   1.715982e-02         0.9349
         50              4.5288   1.621576e-02         0.9450
         51              4.5288   1.527343e-02         0.9419
         52              4.5288   1.438522e-02         0.9418
         53              4.5288   1.351747e-02         0.9397
         54              4.5288   1.264552e-02         0.9355
         55              4.5288   1.153628e-02         0.9123
         56              4.5288   1.046056e-02         0.9068
         57              4.5288   9.460004e-03         0.9043
         58              4.5288   8.685592e-03         0.9181
         59              4.5288   8.173298e-03         0.9410
         60              4.5288   7.713984e-03         0.9438
         61              4.5288   7.156656e-03         0.9278
         62              4.5288   6.530942e-03         0.9126
         63              4.5288   5.991247e-03         0.9174
         64              4.5288   5.501422e-03         0.9182
         65              4.5288   5.160612e-03         0.9381
         66              4.5288   4.843384e-03         0.9385
         67              4.5288   4.559390e-03         0.9414
         68              4.5288   4.290216e-03         0.9410
         69              4.5288   4.035412e-03         0.9406
         70              4.5288   3.819968e-03         0.9466
         71              4.5288   3.611904e-03         0.9455
         72              4.5288   3.419467e-03         0.9467
         73              4.5288   3.208093e-03         0.9382
         74              4.5288   2.993778e-03         0.9332
         75              4.5288   2.726845e-03         0.9108
         76              4.5288   2.457223e-03         0.9011
         77              4.5288   2.225714e-03         0.9058
         78              4.5288   2.033611e-03         0.9137
         79              4.5288   1.914184e-03         0.9413
         80              4.5288   1.799697e-03         0.9402
         81              4.5288   1.670334e-03         0.9281
         82              4.5288   1.522858e-03         0.9117
         83              4.5288   1.388566e-03         0.9118
         84              4.5288   1.275700e-03         0.9187
         85              4.5288   1.194108e-03         0.9360
         86              4.5288   1.119639e-03         0.9376
         87              4.5288   1.061499e-03         0.9481
         88              4.5288   1.000335e-03         0.9424
         89              4.5288   9.452505e-04         0.9449
         90              4.5288   8.985699e-04         0.9506
         91              4.5288   8.504361e-04         0.9464
         92              4.5288   8.098885e-04         0.9523
         93              4.5288   7.613204e-04         0.9400
         94              4.5288   7.080914e-04         0.9301
         95              4.5288   6.480337e-04         0.9152
         96              4.5288   5.812936e-04         0.8970
         97              4.5288   5.259771e-04         0.9048
         98              4.5288   4.851439e-04         0.9224
         99              4.5288   4.556539e-04         0.9392
        100              4.5288   4.293960e-04         0.9424
        101              4.5288   4.009284e-04         0.9337
        102              4.5288   3.650740e-04         0.9106
        103              4.5288   3.336130e-04         0.9138
        104              4.5288   3.087990e-04         0.9256
        105              4.5288   2.887910e-04         0.9352
        106              4.5288   2.724659e-04         0.9435
        107              4.5288   2.612934e-04         0.9590
        108              4.5288   2.473665e-04         0.9467
        109              4.5288   2.350677e-04         0.9503
        110              4.5288   2.251407e-04         0.9578
        111              4.5288   2.138501e-04         0.9499
        112              4.5288   2.055829e-04         0.9613
        113              4.5288   1.951170e-04         0.9491
        114              4.5288   1.826712e-04         0.9362
        115              4.5288   1.693517e-04         0.9271
        116              4.5288   1.546645e-04         0.9133
        117              4.5288   1.414307e-04         0.9144
        118              4.5288   1.325003e-04         0.9369
        119              4.5288   1.265273e-04         0.9549
        120              4.5288   1.208508e-04         0.9551
        121              4.5288   1.144998e-04         0.9474
        122              4.5288   1.060693e-04         0.9264
        123              4.5288   9.925553e-05         0.9358
        124              4.5288   9.370971e-05         0.9441
        125              4.5288   8.940171e-05         0.9540
        126              4.5288   8.594241e-05         0.9613
        127              4.5288   8.353282e-05         0.9720
        128              4.5288   8.024825e-05         0.9607
        129              4.5288   7.731457e-05         0.9634
        130              4.5288   7.480387e-05         0.9675
        131              4.5288   7.216509e-05         0.9647
        132              4.5288   7.037042e-05         0.9751
        133              4.5288   6.794323e-05         0.9655
        134              4.5288   6.514613e-05         0.9588
        135              4.5288   6.205491e-05         0.9525
        136              4.5288   5.863003e-05         0.9448
        137              4.5288   5.502975e-05         0.9386
        138              4.5288   5.254025e-05         0.9548
        139              4.5288   5.086614e-05         0.9681
        140              4.5288   4.932885e-05         0.9698
        141              4.5288   4.746658e-05         0.9622
        142              4.5288   4.485311e-05         0.9449
        143              4.5288   4.297201e-05         0.9581
        144              4.5288   4.136565e-05         0.9626
        145              4.5288   4.015194e-05         0.9707
        146              4.5288   3.900880e-05         0.9715
        147              4.5288   3.820714e-05         0.9794
        148              4.5288   3.706657e-05         0.9701
        149              4.5288   3.596322e-05         0.9702
        150              4.5288   3.497656e-05         0.9726
        151              4.5288   3.398798e-05         0.9717
        152              4.5288   3.333172e-05         0.9807
        153              4.5288   3.242272e-05         0.9727
        154              4.5288   3.145855e-05         0.9703
        155              4.5288   3.036427e-05         0.9652
        156              4.5288   2.914779e-05         0.9599
        157              4.5288   2.764016e-05         0.9483
        158              4.5288   2.659135e-05         0.9621
        159              4.5288   2.590356e-05         0.9741
        160              4.5288   2.516156e-05         0.9714
        161              4.5288   2.434308e-05         0.9675
        162              4.5288   2.309417e-05         0.9487
        163              4.5288   2.224793e-05         0.9634
        164              4.5288   2.147848e-05         0.9654
        165              4.5288   2.090239e-05         0.9732
        166              4.5288   2.035908e-05         0.9740
        167              4.5288   1.995579e-05         0.9802
        168              4.5288   1.940278e-05         0.9723
        169              4.5288   1.886533e-05         0.9723
        170              4.5288   1.835445e-05         0.9729
        171              4.5288   1.786542e-05         0.9734
        172              4.5288   1.753188e-05         0.9813
        173              4.5288   1.706333e-05         0.9733
        174              4.5288   1.659876e-05         0.9728
        175              4.5288   1.604605e-05         0.9667
        176              4.5288   1.542272e-05         0.9612
        177              4.5288   1.468205e-05         0.9520
        178              4.5288   1.418284e-05         0.9660
        179              4.5288   1.381454e-05         0.9740
        180              4.5288   1.343885e-05         0.9728
     ----------------------------------------------------------------------
     Total Iterations: 181
     Avg Convergence Rate:               0.9265
     Final Residual:           1.343885e-05
     Total Reduction in Residual:      9.926137e-07
     Maximum Memory Usage:                4.529 GB
     ----------------------------------------------------------------------

Total Time: 7.59087 setup: 0.583601 s solve: 7.00727 s solve(per iteration): 0.0387142 s

  1. Thats why I tried with ILU smoother to see if it can improve the performance (Using FSAI as complex smoother in HYPRE helped in my case) and when I found that all the configuration involving ILU did not work for my matrix, so I wonder if there may be additional info needed for Multicolor ILU. Here is the output I got when using this configuration (based on GMRES_AMG_D2.json but just change smoother to ILU) { "config_version": 2, "determinism_flag": 1, "exception_handling" : 1, "solver": { "print_grid_stats": 1, "store_res_history": 1, "solver": "GMRES", "print_solve_stats": 1, "obtain_timings": 1, "preconditioner": { "interpolator": "D2", "print_grid_stats": 1, "solver": "AMG", "smoother": "MULTICOLOR_DILU", "presweeps": 2, "selector": "PMIS", "coarsest_sweeps": 2, "coarse_solver": "NOSOLVER", "max_iters": 1, "interp_max_elements": 4, "min_coarse_rows": 2, "scope": "amg_solver", "max_levels": 24, "cycle": "V", "postsweeps": 2 }, "max_iters": 100, "monitor_residual": 1, "gmres_n_restart": 10, "convergence": "RELATIVE_INI_CORE", "tolerance": 1e-06, "norm": "L2" } } Here is the output: AMGX version 2.5.0 Built on Apr 25 2024, 13:20:11 Compiled with CUDA Runtime 12.4, using CUDA driver 12.4 AMG Grid: Number of Levels: 9 LVL ROWS NNZ PARTS SPRSTY Mem (GB)

       0(D)      1507876         113305856      1  4.98e-05            1.3
       1(D)       281692          31103364      1  0.000392          0.706
       2(D)        39859           4883517      1   0.00307          0.111
       3(D)         4793            328295      1    0.0143        0.00753
       4(D)          762             35364      1    0.0609       0.000822
       5(D)          198              7160      1     0.183       0.000168
       6(D)           60              1622      1     0.451       3.88e-05
       7(D)           18               254      1     0.784       6.45e-06
       8(D)            6                36      1         1       9.98e-07
     ----------------------------------------------------------------------
     Grid Complexity: 1.21712
     Operator Complexity: 1.3209
     Total Memory Usage: 2.12977 GB
     ----------------------------------------------------------------------
       iter      Mem Usage (GB)       residual           rate
     ----------------------------------------------------------------------
        Ini             4.52295   1.353886e+01
          0             4.52295   1.353814e+01         0.9999
          1              4.5229   1.353091e+01         0.9995
          2              4.5229   1.353089e+01         1.0000
          3              4.5229   1.353076e+01         1.0000
          4              4.5229   1.353076e+01         1.0000
          5              4.5229   1.351923e+01         0.9991
          6              4.5229   1.351887e+01         1.0000
          7              4.5229   1.350736e+01         0.9991
          8              4.5229   1.350569e+01         0.9999
          9              4.5229   1.350562e+01         1.0000
         10              4.5229   1.350562e+01         1.0000
         11              4.5229   1.350562e+01         1.0000
         12              4.5229   1.350562e+01         1.0000
         13              4.5229   1.350562e+01         1.0000
         14              4.5229   1.350562e+01         1.0000
         15              4.5229   1.350562e+01         1.0000
         16              4.5229   1.350562e+01         1.0000
         17              4.5229   1.350562e+01         1.0000
         18              4.5229   1.350562e+01         1.0000
         19              4.5229   1.350555e+01         1.0000
         20              4.5229   1.350555e+01         1.0000
         21              4.5229   1.350555e+01         1.0000
         22              4.5229   1.350555e+01         1.0000
         23              4.5229   1.350555e+01         1.0000
         24              4.5229   1.350555e+01         1.0000
         25              4.5229   1.350555e+01         1.0000
         26              4.5229   1.350555e+01         1.0000
         27              4.5229   1.350555e+01         1.0000
         28              4.5229   1.350555e+01         1.0000
         29              4.5229   1.350552e+01         1.0000
         30              4.5229   1.350552e+01         1.0000
         31              4.5229   1.350552e+01         1.0000
         32              4.5229   1.350552e+01         1.0000
         33              4.5229   1.350552e+01         1.0000
         34              4.5229   1.350552e+01         1.0000
         35              4.5229   1.350552e+01         1.0000
         36              4.5229   1.350552e+01         1.0000
         37              4.5229   1.350552e+01         1.0000
         38              4.5229   1.350552e+01         1.0000
         39              4.5229   1.350551e+01         1.0000
         40              4.5229   1.350551e+01         1.0000
         41              4.5229   1.350551e+01         1.0000
         42              4.5229   1.350551e+01         1.0000
         43              4.5229   1.350551e+01         1.0000
         44              4.5229   1.350551e+01         1.0000
         45              4.5229   1.350551e+01         1.0000
         46              4.5229   1.350551e+01         1.0000
         47              4.5229   1.350551e+01         1.0000
         48              4.5229   1.350551e+01         1.0000
         49              4.5229   1.350551e+01         1.0000
         50              4.5229   1.350551e+01         1.0000
         51              4.5229   1.350551e+01         1.0000
         52              4.5229   1.350551e+01         1.0000
         53              4.5229   1.350551e+01         1.0000
         54              4.5229   1.350551e+01         1.0000
         55              4.5229   1.350551e+01         1.0000
         56              4.5229   1.350551e+01         1.0000
         57              4.5229   1.350551e+01         1.0000
         58              4.5229   1.350551e+01         1.0000
         59              4.5229   1.350551e+01         1.0000
         60              4.5229   1.350551e+01         1.0000
         61              4.5229   1.350551e+01         1.0000
         62              4.5229   1.350551e+01         1.0000
         63              4.5229   1.350551e+01         1.0000
         64              4.5229   1.350551e+01         1.0000
         65              4.5229   1.350551e+01         1.0000
         66              4.5229   1.350551e+01         1.0000
         67              4.5229   1.350551e+01         1.0000
         68              4.5229   1.350551e+01         1.0000
         69              4.5229   1.350551e+01         1.0000
         70              4.5229   1.350551e+01         1.0000
         71              4.5229   1.350551e+01         1.0000
         72              4.5229   1.350551e+01         1.0000
         73              4.5229   1.350551e+01         1.0000
         74              4.5229   1.350551e+01         1.0000
         75              4.5229   1.350551e+01         1.0000
         76              4.5229   1.350551e+01         1.0000
         77              4.5229   1.350551e+01         1.0000
         78              4.5229   1.350551e+01         1.0000
         79              4.5229   1.350551e+01         1.0000
         80              4.5229   1.350551e+01         1.0000
         81              4.5229   1.350551e+01         1.0000
         82              4.5229   1.350551e+01         1.0000
         83              4.5229   1.350551e+01         1.0000
         84              4.5229   1.350551e+01         1.0000
         85              4.5229   1.350551e+01         1.0000
         86              4.5229   1.350551e+01         1.0000
         87              4.5229   1.350551e+01         1.0000
         88              4.5229   1.350551e+01         1.0000
         89              4.5229   1.350551e+01         1.0000
         90              4.5229   1.350551e+01         1.0000
         91              4.5229   1.350551e+01         1.0000
         92              4.5229   1.350551e+01         1.0000
         93              4.5229   1.350551e+01         1.0000
         94              4.5229   1.350551e+01         1.0000
         95              4.5229   1.350551e+01         1.0000
         96              4.5229   1.350551e+01         1.0000
         97              4.5229   1.350551e+01         1.0000
         98              4.5229   1.350551e+01         1.0000
         99              4.5229   1.350551e+01         1.0000
     ----------------------------------------------------------------------
     Total Iterations: 100
     Avg Convergence Rate:               1.0000
     Final Residual:           1.350551e+01
     Total Reduction in Residual:      9.975368e-01
     Maximum Memory Usage:                4.523 GB
     ----------------------------------------------------------------------

    Total Time: 40.3522 setup: 1.15667 s solve: 39.1955 s

When I tried to used PCG_ILU.json, it return nan.

AMGX version 2.5.0 Built on Apr 25 2024, 13:20:11 Compiled with CUDA Runtime 12.4, using CUDA driver 12.4 iter Mem Usage (GB) residual rate

        Ini                   0   1.353886e+01
          0                   0            nan            nan
          1              0.0000      -nan(ind)      -nan(ind)
          2              0.0000            nan            nan
          3              0.0000      -nan(ind)      -nan(ind)
          4              0.0000            nan            nan
          5              0.0000      -nan(ind)      -nan(ind)
          6              0.0000            nan            nan
          7              0.0000      -nan(ind)      -nan(ind)
          8              0.0000            nan            nan
          9              0.0000      -nan(ind)      -nan(ind)
         10              0.0000            nan            nan
         11              0.0000      -nan(ind)      -nan(ind)
         12              0.0000            nan            nan
         13              0.0000      -nan(ind)      -nan(ind)
         14              0.0000            nan            nan
         15              0.0000      -nan(ind)      -nan(ind)
         16              0.0000            nan            nan
         17              0.0000      -nan(ind)      -nan(ind)
         18              0.0000            nan            nan
         19              0.0000      -nan(ind)      -nan(ind)
     ----------------------------------------------------------------------
     Total Iterations: 20
     Avg Convergence Rate:            -nan(ind)
     Final Residual:              -nan(ind)
     Total Reduction in Residual:         -nan(ind)
     Maximum Memory Usage:                0.000 GB
     ----------------------------------------------------------------------

Total Time: 92.0629 setup: 0.282855 s solve: 91.78 s

Basically I am still struggling to get the right configuration for the matrix. The best results I got is from CG without any preconditioner. I hope this should clarify my issues.