search

ROCm / hipDNN

A thin wrapper around miOpen and cuDNN
36 stars 15 forks source link

SaveAsPriorBuffer is not working if dData is offset of hipMalloced buffer #48

Open eddy16112 opened 4 years ago

eddy16112 commented 4 years ago

In https://github.com/ROCmSoftwarePlatform/hipDNN/blob/master/library/src/hcc_detail/hipdnn_miopen.cpp#L105

    CHECK_HIP(hipMemPtrGetInfo(
        dData, &dPriorSize));  // Get the info of the gradient dx size

If the dData is not directly allocated by hipMalloc, instead, it is part of a pre-allocated memory pool, the dPriorSize obtained is not correct. I think the implementation of SaveAsPriorBuffer only works if dData is allocated by hipMalloc.

Delaunay commented 4 years ago

Yes, it looks like hipMalloc has some sort of memory tracking capabilities. So if you use a pre-allocated memory pool the tracking is not aware of how your pool was chunked and cannot retrieve the size of the buffer.

https://github.com/ROCm-Developer-Tools/HIP/blob/45b73e0961ff2e5e39ae7888dec145ed1c74d7d2/src/hip_memory.cpp#L69

eddy16112 commented 4 years ago

Correct. I think it is better to support memory pool as many people use it to reduce the cost of malloc. By the way, I think there is a performance issue about dPrior, as it is allocated by hipMalloc and freed later every time.