A Tensor Library

[12345swordy]

As discussed in discord having a basic Tensor library that would make basic acceleration for 1d arrays, 2d matrices and 3d+ tensors a lot easier to use. As of right now, it is very tedious as you have to manually allocate memory for the gpu to use and you have to create the kernals programs. Having a library that abstracts the complicity away, so that the programmer can just focus on the problem on hand would be a major blessing for us using the library.

Similar to pytorch tensor library https://pytorch.org/docs/stable/tensors.html Other libraries mentioned at discord: https://www.nuget.org/packages/System.Numerics.Tensors/8.0.0-rc.2.23479.6

[MPSQUARK]

Hi, I'm one of the members of the ilgpu community, and I am developing a library that hopes to fill this void. As you have rightly pointed out, ilgpu is great as it offers the ability to accelerate code, however working with it requires a lot of knowledge into the functioning of the hardware (GPUs) and a lot of micromanagement. I.e. copying data back and fourth, syncing, writing kernels etc. The library I am developing, BAVCL, is a wrapper library around ILGPU, and hopes to offer a plug and play API which uses ilgpu behind scenes (in combo with a bit of simd and other tricks) for working with Vectors, matrices and other Array like structure. There's already a fair bit if functionality in the library, including a thread safe gpu memory management solution, and lot of useful functions for vector math.

The idea behind the app is to offer a similar experience to numpy/scipy but in C#. The library is in development so for example 3d+ data structures and their math has only been conceptualised so far, and matrix math needs more functionality however a large portion of things is already implemented. I work closely with the ilgpu community, and do my best to keep up to date with the latest ilgpu news.

Another important factor leading me to develop the library is to reduce the steepness of the learning curve, ilgpu is a huge product with a tonne of cool features but sometimes all you want is to get an accelerator and do A X B + C a few billion times. Which in bavcl is as simple as GPU gpu = new(); Vector vec1 = ... Define vectors Vec1 X vec2 + vec3 And all the memory management, the loading, unloading of data, math, kernels all gets handled automatically. 😀

If you have any questions I am available in the ilgpu discord, as MPaw7. And the library is BAVCL.

[12345swordy]

Hi, I see the library that you are developing and the major issue that I quickly found is the lack of nuget support. Are you planning to add nuget support as soon as the initial stable release has come out?

[MPSQUARK]

There have been plans to add nuget support. Possibly in the very near future seeing as I consider the lastest versions in git to be stable. Currently major releases have been released as releases on git. Apart from that the master branch should be stable for use in dev when using the library from git, as any new features that could break functionality are developed in development branches and merged in when safe to do so.

[12345swordy]

Just to be clear on this, I am still working on tensor library on my end as I can't wait for your library to release on nuget.

[computablee]

A problem with tensor library support that I see is compatibility between CUDA and ROCm. For instance, CUDA doesn't support FP64 WMMA instructions, but ROCm's matrix cores do support FP64 WMMA (at least on the cards I've used). You don't want to just go for the maximum intersection of features, because then you omit a lot of nice-to-haves for AMD GPU users, but at the same time you want to be clear about performance restrictions. Do you throw a warning if an NVIDIA user tries to do FP64 GEMM that the runtime is falling back to non-tensor-core execution? Do you forbid it at all? Lots of difficult questions.

[12345swordy]

There is a CPU base tensor library out by MS that can be used as a reference. https://www.nuget.org/packages/System.Numerics.Tensors

[HowardvanRooijen]

And .NET 9.0 will have a new Tensor type: https://github.com/dotnet/core/blob/main/release-notes/9.0/preview/preview4/libraries.md#new-tensort-type building on System.Numerics.Tensors