Tpetra: performance regression in CrsGraph filling and fillComplete

[brian-kelley]

Bug Report

In #12009, graph filling and fillComplete both slowed down significantly, even though the changes should have not had an effect on those. Here is some data from the 4-rank FE assembly performance test on vortex.

Before (bbe586f):

<timing name="VortexBMK: Tpetra FE Assembly 4 ranks" value="35.1719">
  <timing name="Kokkos::deep_copy [Host=&gt;Cuda]" value="0.0579755"/>
  <timing name="Kokkos::deep_copy [Cuda=&gt;Host]" value="0.240188"/>
  <timing name="Kokkos::deep_copy [Host=&gt;Host]" value="0.0364796"/>
  <timing name="1) ElementLoop  (Graph)" value="20.8664"/>
  <timing name="2) FillComplete (Graph)" value="12.9501">
    <timing name="Kokkos::deep_copy [Host=&gt;Cuda]" value="0.487526"/>
    <timing name="Kokkos::deep_copy [Cuda=&gt;Cuda]" value="2.2446e-05"/>
    <timing name="Kokkos::deep_copy [Cuda=&gt;Host]" value="0.373551"/>
    <timing name="Kokkos::deep_copy [Host=&gt;Host]" value="1.4665e-05"/>
    <timing name="Remainder" value="12.089"/>
  </timing>

After (ca4f21a):

<timing name="VortexBMK: Tpetra FE Assembly 4 ranks" value="39.5245">
  <timing name="Kokkos::deep_copy [Host=&gt;Cuda]" value="0.0592174"/>
  <timing name="Kokkos::deep_copy [Cuda=&gt;Host]" value="0.239556"/>
  <timing name="Kokkos::deep_copy [Host=&gt;Host]" value="0.036638"/>
  <timing name="1) ElementLoop  (Graph)" value="23.6164"/>
  <timing name="2) FillComplete (Graph)" value="14.5538">
    <timing name="Kokkos::deep_copy [Host=&gt;Cuda]" value="0.49985"/>
    <timing name="Kokkos::deep_copy [Cuda=&gt;Cuda]" value="2.07708e-05"/>
    <timing name="Kokkos::deep_copy [Cuda=&gt;Host]" value="0.411824"/>
    <timing name="Kokkos::deep_copy [Host=&gt;Host]" value="1.4499e-05"/>
    <timing name="Remainder" value="13.6421"/>
  </timing>

So the total time went up by over 4 seconds. This slowdown also happened on eclipse (CTS1).

@cwpearson @csiefer2

[github-actions[bot]]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity. If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label. If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE. If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.