search

tonybaloney / CSnakes

https://tonybaloney.github.io/CSnakes/
MIT License
270 stars 20 forks source link

Is c++/cli gcnew "magic" from a CPython module possible? Or a module written in C# ? #251

Open minesworld opened 1 week ago

minesworld commented 1 week ago

The idea is to use existing C# and/or new highly specialized classes written in c++ to move data from CPython to C# . Why waste time and heap by wrapping if the c# object can be created directly from within a highly specialized CPython module?

If that is possible - could someone for noobs like me provide a Visual Studio code project?

In the projects I've written the count of objects from passed Python to C# was about 100-1000 times more than the other way... this time it might be 50000 X upwards. As it will be a WinUI3 application accessing wrapped CPython object attributes would be death by roundtrips... . Last resort would be using a shared SQLite instance on both sides...

tonybaloney commented 1 week ago

Look at the buffer protocol implementation. I think this is what you're looking for https://tonybaloney.github.io/CSnakes/buffers/

That works for numpy, I haven't tried pandas, but it'll work with bytes and bytearray and in future versions of Python str will implement buffer protocol.

It won't get any faster than this, we're effectively casting the internal memory structure of the Python object into a C# Span (which is a fancy pointer)

tonybaloney commented 1 week ago

Checkout this demo function https://github.com/tonybaloney/CSnakes/blob/main/samples/simple/QuickConsoleTest/Program.cs#L54-L74 for an example

minesworld commented 1 week ago

Thanks, using buffers is a brilliant idea, I just have to get my "head around" it and find out what the limits are to such an approach while using it with dotnet/CLR and WinUI3 ...

With the idea behind your CSnakes it seems to be possible to allocate the needed chunk of memory on the python side, fill it as a struct which could be used from C#, having an interface/class code generated, compile that at runtime if needed and use that as a super efficiant bridge from python to C#.

If it is possible to construct dotnet string using pointers to UTF-16 data there would be two options: the "clean" one by putting the UTF-16 data into python buffer too or giving pointers directly to the PyString UTF-16 data and dealing with those reference counts on the C# side too.

Hope I find out what is possible. For me CSnakes looks like the most usefull and performant dotnet bridge there is today.