search

ARM-software / ComputeLibrary

The Compute Library is a set of computer vision and machine learning functions optimised for both Arm CPUs and GPUs using SIMD technologies.
MIT License
2.81k stars 774 forks source link

sparse gemm kernels are not supported in ACL #1084

Open snadampal opened 9 months ago

snadampal commented 9 months ago

Output of 'strings libarm_compute.so | grep arm_compute_version': arm_compute_version=v23.11 Build options: {'Werror': '0', 'debug': '0', 'neon': '1', 'opencl': '0', 'embed_kernels': '0', 'os': 'linux', 'arch': 'armv8a', 'build': 'native', 'multi_isa': '1', 'fixed_format_kernels': '1', 'openmp': '1', 'cppthreads': '0'} Git hash=b'add70ace1e57f65d1ae4d0cedaec6e4578cf87ff'

Platform: AWS c7g.16xl

Operating System: Ubuntu 22.04

Problem description:

PyTorch supports sparse tensor formats The request is to provide aarch64 gemm kernels that accept these sparse formatted tensors and does sparse gemm implementation to achieve better performance.

morgolock commented 8 months ago

Hi @snadampal

Thanks for raising this. We will discuss the feature request with the team.

morgolock commented 8 months ago

Hi @snadampal

We discussed this with the team, we are considering exploring sparse tensors support in the context of GenAI but this is not officially in the roadmap for ACL. There are no plans to implement this feature.

We would be interested in specific use cases for ACL which you could share with us.

Hope this helps.

snadampal commented 6 months ago

Hi @morgolock , isn't ACL targeted for GenAI use cases? My requirement is mainly to accelerate sparse LLMs inference with ACL gemm kernels. Are you planning a different GEMM library for GenAI ?

morgolock commented 6 months ago

Hi @snadampal

Apologies if I was not clear. we're exploring ways to accelerate GenAI workloads with ACL. This means that we may consider exploring sparse tensors support in ACL to accelerate these workloads in the future but we don't have any work planned for this feature in our roadmap.

My requirement is mainly to accelerate sparse LLMs inference with ACL gemm kernels.

Could you please share more details about the models you would like to accelerate?

Hope this helps

snadampal commented 6 months ago

Hi @morgolock , thanks for the clarification. will share the details with you.

Arnav0400 commented 3 weeks ago

Hi @morgolock, any updates on the plan for sparse GEMM kernels in ACL? This is particularly interesting as sparse LLMs are able to match performance of dense counterparts on task-specific applications. For ref. - https://arxiv.org/pdf/2310.06927

morgolock commented 1 week ago

Hi @Arnav0400

There is no work planned to implement this feature at the moment. I'll discuss it again with the team.