[Clarification Needed] Accessing linear systems data already stored on GPU(s) in rocThrust data structures

[klausbu]

Hi, for clarification:

Is it possible to use rocALUTION to solve a linear system that's already on the GPU stored in rocThrust data structures. The application is based on the concept to upload the data for a simulation in timestep one, keep it there throughout the simulation (hence there are no intermediate device - host - device data transfers) and manipulate the data using rocThrust, only the final result of the simulation is moved back to the host after hundreds or thousands of timesteps. At some point of the on GPU manipulations using rocThrust, linear systems need to be solved which could probably be faster using rocALUTION due to it's potentially better preconditioners if the on GPU data could be accessed with rocALUTION.

Would this be feasible?

[ntrost57]

Hi, you can pass raw (device) data pointers to rocalution using e.g. SetDataPtrCSR(). Once computation has finished, you can get back these data pointers using e.g. LeaveDataPtrCSR(). If your raw data pointers are on the GPU, you will have to move the (empty) rocalution object to the GPU, before filling the pointers. Similarly, if you LeaveDataPtrCSR() from an rocalution GPU object, the obtained pointers will be valid on the GPU only. Effectively, there will be no host computation or transfers involved. Does this answer your question?

[ntrost57]

Here is some code fragment:

double* dev_data;
hipMalloc(&dev_data, sizeof(double) * size);

LocalVector<double> rocalution_vec;

// Move the empty rocalution_vec to GPU, such that rocalution expects a GPU pointer
rocalution_vec.MoveToAccelerator();

// Set the raw data pointer
rocalution_vec.SetDataPtr(&dev_data, "my vector", size);

// Be aware, that dev_data is pointing to NULL after calling SetDataPtr

// Do some work with rocalution
// ...

// Once done, get your data back. rocalution_vec is still a device vector here. If you want it as a host pointer, you can
// achieve this by calling MoveToHost().
rocalution_vec.LeaveDataPtr(&dev_vec);

// Now, dev_vec is valid again, but rocalution_vec will be empty.

hipFree(dev_vec);

[klausbu]

Once I finished hipifying, I'll look deeper into it, it clarifies a lot.

thank you!

[ntrost57]

Closing this issue for now. Please re-open if things do not work as expected.