AlexandreEichenberger
commented
3 years ago @chentong319 Thanks for this very detailed investigation.
Just to be clear, the slowdown above are using the current implementation with Dynamic vs Static vector shape, right?
It would also be interesting to learn why the later ops get increasingly slower? Are they increasingly larger?
From having looked at assembly, even the simplest loop:
for(int I=0; I<64; I++) a[I] = b[I] + c[I]If unrolled by the backend, there is literally no scheduling of the reads (b and c) past any write (a) because memory disambiguation is quite broken once we are out of MLIR. For this particular example, the only way to move read past write operations was to hav MLIR unroll the loop.
With static sizes, we agree that memory bundling can reuse the memory used for temp tensors quite well. Say this is hard for arbitrary dynamic sizes, the cost of having malloc/free just before every op should technically have nearly the same footprint as the static size (at least same order of magnitude) and I don't believe the overhead associated with malloc/free should be too high if the amount of computations (such as matmul/conv) are nontrivial, as computations are at least O(N3) compared to O(1) of malloc/free costs.
Right now, our policy is to place malloc/free at beginning/end of function. That could be parametrized to before/after op in presence of dynamic tensors. Or malloc/free could be moved toward begin/end of functions vs begin/end of operation depending on static/dynamic tensors.
tungld
chentong319
The argument tensor of yolo model is of shape tensor<?x?x?x3xf32> (called dynamic version here). If we change the shape of the argument with the shape of the input tensor used for inference, tensor<1x416x416x3xf32>, the performance (called static version) is much better (more than x100). I did so investigation into this issue. First, I added instrumentation for the popular op in yolo model: Conv and Mul. Here are the observations:
First difference between dynamic and static version is memory usage. In the static version, the virtual memory size is around 400 m for different versions of model (I tried to remove some operations). For dynamic model, memory usage increases with the number of ops. With only 1/3 of the operations in the original model, the memory size is 22 G. I am not clear about the impact on performance, cold cache misses, unnecessary cache write back and even extra page fault. Can we run some tool to get the details?
For the operations, now I focused on Mul, which is quite simple. For the EmitLLVMIR, the major difference is that the extra get dim size operations inside loop body. If the compiler does not know the alias info, it is possible that these operations are not moved to out side of loop, introducing extra load operations. In next step, we need to check the bc file to verify what the final code is. How to generate bc file?