asroy
commented
4 years ago
- Col2Im and Im2Col must be fully dynamic and placed in the binary cache, to ensure launching gemm solution without any overhead. @asroy
We can implement Col2Im and Im2Col using dynamic composable kernels, once it is implemented. Converting the OCL kernel will be throw-away work.
Non-convolution operations can also be written using dynamic composable kernels.
We can have dynamic composable kernels for following operations in ROCm 3.10 time frame: iGEMM, Im2Col, Col2Im. Pooling is like a simplified version of convolution, so it's also straightforward to rewritten using dynamic composable kernel, but since it's already fully dynamic, so we don't need to do it for ROCm 3.10.
Batch-norm is performance critical, so it may need some optimization work, and can be rewritten using composable kernel after 3.10.
atamazov
JehandadKhan
t-vi
hgaspar
This issue describes the initial ideas for the overall plan and opens it for discussion. While issue #337 explains some of the details, however, it is more focused on assigning dynamic kernels to developers and lacks the overall picture.
Primary Challenge
The primary issue facing MIOpen right now is that the cost of compiling and searching for the optimum algorithm far outweighs the benefits in performance we get by doing such an exhaustive search. While such an approach does make some sense for classical deep networks where the same back bone network is compiled once and then launched many times. However, if the parameters in the network are data dependent many kernels might be compiled while the realized performance advantage is very little in comparison to the amount of time spent on compiling and benchmarking kernels.
The primary thesis presented here is that for a convolution config that MIOpen does not know we should not compile any kernel at all and must be served by a precompiled, parameter-less kernel ( such as Winograd, Dynamic iGEMM or a rocBlas call ). This ensures that no time is spent on compiling a kernel and the GPU may be kept busy all the time.
It may be argued that an assembly kernel has a much shorter assembly/finalizing time and therefore, it may be acceptable to add asm kernels to the mix above. However, even an asm kernel takes more than 100ms to finalize, even which may be too long a time to benefit from the improved performance. I am planning an experiment to prove/disprove this theory.
Even though the above issues have been brought to the forefront by Mask-RCNN and retina-net, MIOpen faces the same issues on any other network which may not be tracked by us, therefore, it is imperative that we develop a solution that is general purpose and not a quick fix.
Proposal for Convolution Operations
The above goal may be achieved by relying on the find-db and the binary cache.
Ensuring that all winning solvers in the find-db are present in the binary cahche package. This would ensure that we maintain performance parity for classic network and special workloads that are critical.
All dynamic kernels ( in their various incarnations) must be part must be part of the binary cache. This is to ensure that when there is a find-db miss the associated binary cache is available and ready to be launched.
Col2Im and Im2Col must be fully dynamic and placed in the binary cache, to ensure launching gemm solution without any overhead. @asroy
A (yet another) find mode which operates as follows (@atamazov):
MIOPEN_FIND_MODE=1) should be allowed to write to the find-db, since everything else is a heuristic and should not be serialized.isWinogradApplicableAndFastbut formalized to be extendable with any logic that dependens on theProblemDescription. This is important for extensibility and future proofing our find-machinery. This goal is lower priority as compared to the other items in the list above but must be kept in mind while designing code paths.What is the difference of Dynamic convolution kernel vs. "non-dynamic"?
batchsize,image_heightandimage_widthare always kept dynamic, other parameters may be made static to improve performance. However, it must be kept in mind that for such kernels all the permutations of all the static arguments would need to be hosted in the binary cache resulting in increased disk usage for the library.Proposal for Non-Convolution Operations
While convolution operations form the bulk of expensive operations, non-convolution operations also contribute considerable amount of delay, especially if kernels follow the same design such that any change in the input parameters require a re-compile. Based on the usage of non-Conv operations in the frameworks Batchnorm is the first priority for conversion to a dynamic kernel since most other ops are implemented internally by higher level frameworks.
For batch-norm we would require three kernels for the following cases ( based on memory access patterns and parameter variations)
These kernels should not have any other compile time specified parameters so we can stash them in the binary cache.
The ASM version of Batch Norm is expected in a few weeks and should hopefully mitigate some of the compilation issues, however, as discussed earlier even the assembly time might not be acceptable in the over all runtime.
It is also possible that we add Batch Norm to the find-db so that we can implement similar mechanisms for non-conv ops as we have for convolution. However, this may be kept as a long term goal due to the amount of work and additional discussion/thought required.
Following Batchnorm, tensor ops would be converted to dynamic kernels ( most of them are already converted as mentioned in issue #337 ) and finally followed by pooling.
Task List:
In addition to the task assignment in dynamic kernels #<> the following tasks have been identified and assigned so far:
Dynamic Im2Col2d / Col2Im2DGuidelines for Reviewers/Comments
Please refrain from referencing comments on other threads, which makes it difficult to follow the conversation. I will try to update the description with discussion below.
@atamazov @daniellowell