Open lintao185 opened 3 months ago
I don't have time right now, but I will experiment with this when I get a chance.
Question: is this code MemoryMarshal.Cast<byte, T>(ndarray.tobytes()); even valid in a .NET standard application? In other words , will it work on .net versions for linux and mac too?
I welcome assistance. If you want branch the code and try to turn this into a generic API for different data types, please fell free. I will review it and possibly accept it into the main branch.
I tried to add a unit test to experiment with your sample. I does not compile cleanly for me. I have not used Span
These lines do not compile cleanly:
values[start..end]; shapeIDims[1..]
Can you help?
[TestMethod]
public void test_lintao185_1()
{
var gn = np.ones((5, 10000, 10000)).astype(np.Float32);
var gnLs = GetList<float>(gn);
}
public static IList GetList<T>(ndarray ndarray) where T : struct
{
var values = MemoryMarshal.Cast<byte, T>(ndarray.tobytes());
var list = GetList(values, 0, values.Length, ndarray.shape.iDims);
return list;
}
private static IList GetList<T>(Span<T> values, int start, int end, long[] shapeIDims)
{
if (shapeIDims.Length == 1)
{
var listr = new List<T>(end - start);
listr.AddRange(values[start..end]);
return listr;
}
var genericType = typeof(List<>);
var argType = typeof(T);
for (int i = 0; i < shapeIDims.Length; i++)
{
argType = genericType.MakeGenericType(argType);
}
var list = (IList)Activator.CreateInstance(argType);
var valueLength = end - start;
var length = (int)(valueLength / shapeIDims[0]);
for (int i = 0; i < shapeIDims[0]; i++)
{
var newStart = start + (i * length);
var newEnd = start + ((i + 1) * length);
newEnd = newEnd >= values.Length ? values.Length : newEnd;
list.Add(GetList(values, newStart, newEnd, shapeIDims[1..]));
}
return list;
}
Hi Kevin,
I just happened to click on this and am interested... so please pardon me for butting in.
The two lines that you mention values[start..end]; shapeIDims[1..]
The compile error is not specific to Spans, Those lines use a feature from c# version 8 called "range", (https://learn.microsoft.com/en-us/dotnet/csharp/tutorials/ranges-indexes)
list.Add(GetList(values, newStart, newEnd, shapeIDims[1..]));
can be read as list.Add(GetList(values, newStart, newEnd, shapeIDims.Skip(1).ToArray()));// Allocates new array
listr.AddRange(values[start..end]);
can be read as listr.AddRange(values.Slice(start,end - start)); //slice() itself does not allocate
With regards to your other question.. Spans and MemoryMarshal.Cast are ".Net standard 2.0" onward and work fine on Linux and all "dotnet" (core). (The implementation of Spans on ".NET Framework 4.8" was slightly different, but it works there too).
A mini primer on Spans:
Memory
if you have some time, (and dont mind watching videos) the following recent presentation does a reasonable job at explaining how revolutionary the introduction of Span is: https://learn.microsoft.com/en-us/shows/on-net/a-complete-dotnet-developers-guide-to-span-with-stephen-toub#time=31m51s
Alex
I appreciate you butting in :) Thanks for the tips. I appreciate you pointing the info.
As I interpret your question, the answer is that YES, converting an existing array to a Span
[TestMethod]
public void test_lintao185_1()
{
var gn = np.ones((100000000)).astype(np.Float32);
System.Diagnostics.Stopwatch sw1 = new System.Diagnostics.Stopwatch();
sw1.Restart();
var bytes1 = gn.tobytes();
var gn1 = np.array(bytes1);
sw1.Stop();
//var gnLs = GetList<float>(gn);
System.Diagnostics.Stopwatch sw2 = new System.Diagnostics.Stopwatch();
sw2.Restart();
var bytes2 = MemoryMarshal.Cast<float, byte>(gn.AsFloatArray());
var gn2 = np.array(bytes2.ToArray());
sw2.Stop();
Console.WriteLine(sw1.ElapsedMilliseconds); <--7743
Console.WriteLine(sw2.ElapsedMilliseconds); <--205
}
In the unit test below, the call to tobytes returns a new array of bytes, copied from the source array. If you modify the bytes1 array it does not affect the original. If you use the MemoryMarshall.Cast method, you get a view into the original array and any modifications made to the bytes2 will be reflected in the original array. So, they are very different.
[TestMethod]
public void test_lintao185_2()
{
var gn = np.ones((10, 10, 10)).astype(np.Int32);
var bytes1 = gn.tobytes();
bytes1[0] = 99;
var gn2 = np.ones((10, 10, 10)).astype(np.Int32);
var bytes2 = MemoryMarshal.Cast<int, byte>(gn2.AsInt32Array());
bytes2[0] = 99;
return;
}
Unless I misunderstood the question, I think the existing implementation of .tobytes() is closer to the python version (although that actually returns a string instead of a byte[]).
But this was a great exercise to show that anyone who wants to map an ndarray object to a different data type can do so much faster with MemoryMarshall.Cast as long as they don't want a copy.
In this example, the conversion is particularly slow.