jinevening
commented
1 year ago Author response draft
We thank the reviewers for their useful comments. This is our response to the questions from the reviewers.
Reviewer A
Q. The DP formulation doesn't take memory limits in consideration. A. Memory limit is not shown in the optimal substructure of DP, but it is in fact considered as our tile size search is performed based on memory footprint estimation. If we cannot find a feasible tile size that meets the memory constraint, that case is ignored during DP algorithm.
Q. What happens if the solution doesn't fit in memory? A. The solution derived by our algorithm can violate the memory constraint, because it is based on the estimated footprint without consideration of fragmentation. In that case we retry compilation with less memory budget to find a solution with less footprint; the last paragraph in Section 3.3 explains it.
Q. In figure 11, what's MV1, MV2, EFN, etc? The benchmarks are not explained. A. They are abbreviations of benchmark models, as shown in Table 2.
Q. Could you please explain why rhs of Figure 13 is best? A. Our description might have confused you. You can see that the outputs of layer groups in Figure 13 (b) are pinned (colored). This indicates that the off-chip transfer of those tensors is removed.
Q. What's v, g, p in DP formula? A. The second paragraph of Section 3.1.3 explains them in detail.
Reviewer B
Q. As to channel expansion. A. The paper's description was based on the assumption that the outputs of intermediate layers are re-computed rather than cached. We will clarify that in the camera-ready version.
Q. How about using Figure 13 as the motivating example instead? A. We agree that Figure 13 shows motivation for joint optimization as well as the benefit of pinning. We'll update the motivating example in the camera-ready version if there is no problem. Thanks for the suggestion.
Q. In Equation (2) better place the max inside the sum over tiles, e.g., to account for some tiles being load-bound while others compute-bound? A. That's a good point. We placed the max outside of the sum, because tiles of the same layer group have similar proportion of load/compute/store, e.g., for all tiles of a layer group, load takes more time than compute or store. We will clarify the point in the final version.
Q. Double-buffering allows loading second ifm in parallel to first compute_1, and loading third ifm needs to wait for first compute_1 to finish using first ifm- rather than wait for first compute_2 to finish? A. At first, we use a unified buffer for different tiles (instead of double buffering) to share a pinned tensor across different tiles. Next, the third ifm is loaded after compute_2, because we pipeline execution of only two tiles to avoid excessive memory pressure. Of course execution of more tiles can be overlapped as you mentioned, with modification in footprint estimation formula.
Q. Equation (4) provides a lower-bound on actual memory allocation because it ignores fragmentation, and only because of that? A. It is difficult to compute "exact" memory footprint as it depends on scheduling, allocation algorithm, and fragmentation. So we use estimated memory footprint during pinning-aware fusion, and do scheduling and allocation after that.
Q. In Algorithm 1, how is the graph topologically sorted - are all sortings examined to select the best one, considering "for v in inputs(V_curr)"? A. We only use a single sorting result. It is not necessary to examine all sortings, because the order of "inputs(V_curr)" does not affect the final result. For example, if there are two inputs v1 and v2, DP algorithm will proceed with v1, and also with v2. The order of v1 and v2 does not affect the final result.
Reviewer C
Q. As to memory footprint estimation of operators that require extra space. A. Some operators consume extra space (e.g., parameters for PRelu), as you mentioned. For that case, the footprint estimation formula must be extended to account for the extra space.
Q. As to the reduction of computation from pinning. A. Pinning can mitigate computation increase from fusion by making fused-layer groups smaller. For example, a fused layer group can be divided into two smaller groups where the intermediate tensor between them is pinned. This reduces the number of consecutive layers in each group, thus reducing duplicate computations. We'll clarify that in the revision.
Q. Is there any challenge to generalize the proposed work to other DNNs than CNN? A. Some RNN models include tensors with dynamic shapes, e.g., tensor size is increased in each iteration. Pinning/fusion of such tensors would be a challenging issue on generalization.
Reviewer D
Q. What is the size of the targeted neural network in the evaluation? A. The sizes of benchmark models are shown in Table 2.
Q. How does the idea of pinning differ from cache locking, at a higher level of abstraction? A. Both cache locking and pinning can reduce off-chip data transfer by manual placement of data on the on-chip memory. A major difference is that pinning is applied to SW(compiler)-controlled SPM on accelerators with strict gate count restrictions, while cache locking is applied to HW-controlled cache which is usually available in richer processors such as CPU. Also, pinning determines placement of data based on latency and memory consumption of DNN processing, while cache locking typically uses execution traces to detect hot data.
Q. What happens when the feature map is too large to fit in the global buffer? A. Our pinning-aware fusion algorithm will automatically apply fusion rather than pinning if the feature map is too large to fit in the global buffer.
Reviewer E Q. Have the authors attempted to solve the problem optimally using ILP? A. We considered to use ILP, but decided not to use it, because our objective function (latency) is not linear in fused-layer execution.
Q. How far can we go if we improve your approach in this direction? A. We put several constraints on our algorithm (e.g., pinning is applied to the whole tensor, and fusion is searched in a forward direction). It is difficult to predict expected improvements, but pinning-aware fusion will definitely find a better solution without such constraints.
Q. What are the limitations of the cost functions revealed by the experimental evaluation? A. Although it did not severely spoil the decisions on pinning/fusion, memory load/store time was fluctuated, which might be related to device workload. We left the workload-aware fusion as a future work.
periannath
parjong
Full comments