search

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
9.05k stars 1.88k forks source link

Decode Date32 arrow type in DataFrame #7261

Open vthemelis opened 1 month ago

vthemelis commented 1 month ago

Is your feature request related to a problem? Please describe. At the moment, it looks like it's not possible to read RecordBatches of Date32s:

using Apache.Arrow;
using Microsoft.Data.Analysis;

var batch = new RecordBatch.Builder()
    .Append("DateColumn", false, new Date32Array.Builder().AppendRange(Enumerable.Repeat(DateTime.Now, 10)).Build())
    .Build();

DataFrame.FromArrowRecordBatch(batch);

gives:

Unhandled exception. System.NotImplementedException: date32
   at Microsoft.Data.Analysis.DataFrame.AppendDataFrameColumnFromArrowArray(Field field, IArrowArray arrowArray, DataFrame ret, String fieldNamePrefix)
   at Microsoft.Data.Analysis.DataFrame.FromArrowRecordBatch(RecordBatch recordBatch)
   at Program.<Main>$(String[] args)