dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
9.02k stars 1.88k forks source link

All DataFrame Elementwise methods uncorrectly work with NULL values #6820

Open asmirnov82 opened 1 year ago

asmirnov82 commented 1 year ago

Describe the bug PrimitiveColumn ElementwiseEquals, ElementwizeNotEquals and other Elementwise method ignore validity information

For example. this unit test fails as it ElementWizeEquals return true for the first item ( as 0 == null):

[Fact]
public void TestElementWiseEquals()
{
    PrimitiveDataFrameColumn<int> intColumn1 = new PrimitiveDataFrameColumn<int>("Int1", new int?[] { 0, 2, 3 });
    PrimitiveDataFrameColumn<int> intColumn2 = new PrimitiveDataFrameColumn<int>("Int2", new int?[] { null, 2, 3 });

    var results = intColumn1.ElementwiseEquals(intColumn2);

    Assert.False(results[0]);
}