Add support for multi-dimensional arrays for model input/output.

[jannickj]

I have a fully working tensorflow model and I litterally just need the last step of having C# run my model, but I am stuck on a null exception.

I have a very simple setup, and I've locked down both sequence length and batch size, however no matter what i do it gives me the exception:

  at Microsoft.ML.Data.TypedCursorable`1.TypedRowBase.<>c__DisplayClass8_0`1.<CreateDirectVBufferSetter>b__0(TRow row)
   at Microsoft.ML.Data.TypedCursorable`1.TypedRowBase.FillValues(TRow row)
   at Microsoft.ML.Data.TypedCursorable`1.RowImplementation.FillValues(TRow row)
   at Microsoft.ML.PredictionEngineBase`2.FillValues(TDst prediction)
   at Microsoft.ML.PredictionEngine`2.Predict(TSrc example, TDst& prediction)
   at MyProject.Model.Run() in 

I have tested that the model works in python and I've made 100% sure the dimensions fit exactly.

public record Features
    {

        [ColumnName("x_1")]
        [VectorType(1, 41, 3)]
        public int[,,] UnigramWindows { get; set; } = null!;
        [ColumnName("x_2")]
        [VectorType(1, 41, 3)]
        public int[,,] BigramWindows { get; set; } = null!;
        [ColumnName("x_3")]
        [VectorType(1, 41, 3)]
        public int[,,] CharTypeWindows { get; set; } = null!;
        [ColumnName("x_4")]
        [VectorType(1, 41, 41)]
        public int[,,] WordsStartingAt { get; set; } = null!;
        [ColumnName("x_5")]
        [VectorType(1, 41, 41)]
        public int[,,] WordsEndingAt { get; set; } = null!;
        [ColumnName("x")]
        [VectorType(1)]
        public int[] SeqLen { get; set; } = null!;
    }

private record Output
{
    [VectorType(1, 41, 6)]
    public float[,,] Identity;
}

private static ITransformer LoadModel(
    MLContext mlContext,
    string modelPath)
{
    var tfModel = mlContext.Model
        .LoadTensorFlowModel(modelPath);
    var schema = tfModel.GetModelSchema();
    var revSchema = schema.Reverse().ToArray();
    var pipeline =
        tfModel
        .ScoreTensorFlowModel(
                outputColumnNames: new[] { "Identity" },
                inputColumnNames:
                new[] {
                    "x",
                    "x_1",
                    "x_2",
                    "x_3",
                    "x_4",
                    "x_5",
                },
                addBatchDimensionInput: false);

    var dataView = mlContext.Data.LoadFromEnumerable(Enumerable.Empty<Features>());
    ITransformer mlModel = pipeline.Fit(dataView);

    return mlModel;
}

public static run() 
{
        var model = LoadModel(mlContext, "model.pb");
    var predictionEngine = mlContext
        .Model
        .CreatePredictionEngine<Features, Output>(model);

        var res = predictionEngine.Predict(features);

    Console.WriteLine(System.Text.Json.JsonSerializer.Serialize(res));
}

[jannickj]

To any unfortunate soul who've had to deal with the same issue, I finally figured it out. 2d Arrays are not supported in dotnet ml, you're supposed to flatten the arrays yourself ><

[michaelgsharp]

@jannickj sorry for the confusion about this. This is something that we are discussing. As we are working on adding TorchSharp support to ML.NET, this will probably become a larger issue, so we are planning on revisiting this and discussing it again in the future.

[jannickj]

I think a simple fix for now would just be to throw an exception that says multidimensional arrays are not supported the problem is null exception makes figuring it out very obscure.

[KonradZaremba]

Ok so just to clarify. If I load tf model to ml.net that needs as input matrix [,] will I get proper output?

[julianogimenez]

Any news on that? I'm workint in a POC for a customer. I have a tensortflow model (N,160,6) but Im not able to input an array. How can I do that?

[jannickj]

@julianogimenez you just have to squeze your multidim array into a single dim array i.e from (N, 160, 6) -> (N 160 6,)