Possibly incorrect preallocation in SNES colouring

[d7919]

I've been running some SD1D cases with the snes solver and I noticed that with snes_view active the output reports that there are > 0 mallocs during MatSetValues. For a problem with ny=200 there are 1200 (nfield * ny) mallocs per call whilst this drops down to something like 6 mallocs per call (per proc presumably) on 50 procs.

Mat Object: 1 MPI processes
      type: seqaij
      rows=1200, cols=1200
      total: nonzeros=21528, allocated nonzeros=24000
      total number of mallocs used during MatSetValues calls=1200

Mat Object: 50 MPI processes
      type: mpiaij
      rows=1200, cols=1200
      total: nonzeros=21564, allocated nonzeros=25218
      total number of mallocs used during MatSetValues calls=6

[bendudson]

Hey @d7919 ! Thanks for this. Which version of the solver was used? I think SD1D currently compiles against an older version of the SNES solver, and there may be some changes. It does look suspicious though that something is not being pre-allocated.

[d7919]

This is the bout next branch of sd1d but updated to the most recent next of bout. I found this changed behaviour quite a bit, with nrhs increasing a fair amount bit overall things are making more reliable progress.

On Wed, 9 Mar 2022, 17:39 Ben Dudson, @.***> wrote:

Hey @d7919 https://github.com/d7919 ! Thanks for this. Which version of the solver was used? I think SD1D currently compiles against an older version of the SNES solver, and there may be some changes. It does look suspicious though that something is not being pre-allocated.

— Reply to this email directly, view it on GitHub https://github.com/boutproject/BOUT-dev/issues/2524#issuecomment-1063186473, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANV5ISFJ35YLONCC6Q4TUTU7DO6ZANCNFSM5QJPHWJA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[bendudson]

One of the main things to change in the SNES solver was the form of the nonlinear equation being solved : https://github.com/boutproject/BOUT-dev/blob/next/src/solver/impls/snes/snes.cxx#L919 Previously the method was backward_euler, whereas now the default is rearranged_backward_euler . Which one is best is probably problem-dependent.

[d7919]

Thanks, I did spot that but when I tried going back to the original form, nrhs increased further! I've managed to get the runtime down quite a lot by tuning the petsc setup so I'm not too concerned currently about the apparent change in nrhs (particularly as it seems more robust now). I thought the memory message may be a sign that something isn't quite as we expect so thought I'd flag that. It's possible this could give a small improvement as well.

On Wed, 9 Mar 2022, 19:14 Ben Dudson, @.***> wrote:

One of the main things to change in the SNES solver was the form of the nonlinear equation being solved : https://github.com/boutproject/BOUT-dev/blob/next/src/solver/impls/snes/snes.cxx#L919 Previously the method was backward_euler, whereas now the default is rearranged_backward_euler . Which one is best is probably problem-dependent.

— Reply to this email directly, view it on GitHub https://github.com/boutproject/BOUT-dev/issues/2524#issuecomment-1063267736, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANV5IRN23O65QXIZJN73V3U7D2A7ANCNFSM5QJPHWJA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>