[Feature]: Support flashdecoding++ heuristic dataflow and profiling optimization

[chizhang118]

• Q3 Collaboration Plan of Infra and IaaS Labs: https://bytedance.us.larkoffice.com/docx/HKXfdRh1noMrbAxcgL2ureGasdQ

• FlashDecoding++ Summary: https://bytedance.larkoffice.com/wiki/WbqXwRL3qi0x18kJVkAcU0HZnte, including Asynchronized Softmax, Double Buffering, Heuristic Dataflow, and more profiling-related optimizations.

• The new kv cache memory layout and performance benchmarks is also needed.

• Inference Improvement Weekly Status and Progress: https://bytedance.us.larkoffice.com/docx/RGnPdj5gfoBN3YxuY3yuLtlQsjb

[chizhang118]

Profling heuristic dataflow: https://github.com/vllm-project/vllm/blob/df845b2b46c3e30f5bd3e3be286285ed148323fc/vllm/worker/worker.py#L248

[chizhang118]

Implemented the flat gemm and support heuristic dataflow with small batch decoding using flat gemm.

https://code.byted.org/inf/dAttention/tree/feat/flat-gemm

[chizhang118]

https://code.byted.org/inf/dAttention/merge_requests/13

[chizhang118]

https://code.byted.org/inf/dAttention/merge_requests/18 Code merged.