Sanity check on Device <--> Host memory transfers on GPU simulations

[sbryngelson]

A recent profile of @anshgupta1234's branch revealed several memory transfers between host and device within a time step. They are short in the grand scheme of things because the case is small, but I could imagine this being problematic otherwise. I am posting some nsight sys screenshots here that should be investigated. Some of them appear to be moving data but are not associated with Ansh's IBM implementation (seemingly).

[sbryngelson]

@anshgupta1234 can you clarify if this profile was run with or without the !$acc parallel loop in the IBM region of the code that we discussed?

[anshgupta1234]

@sbryngelson it was without the !$acc parallel loop

[sbryngelson]

Can you add the acc loop back in, run it with 1M grid cells in 2D for 10 time steps, then send me the nsys profile again? @anshgupta1234

[sbryngelson]

Closing this, the compute sections look fine in the new profile. Any stray uploads are just local variables, they only appear important because the IBM update takes so little time but in practice they do not matter.