Struggling with empty arrays

[JasonBoggsAtHMSdotCom]

I'm struggling with an issue checking for properties in empty arrays. When I use .Any(predicate) against an empty array, I get

The binary operator Equal is not defined for the types 'System.Object' and 'System.Int32'.

In the code snippet below, I'm using the rule: things.Any(a == 1) for the following payloads: { "things": [] } - expect True, receiving False and exception ❌ { "things": [ { "a": 1 } ] } - expect False ✔️ { "things": [ { "a": 2 } ] } - expect True ✔️

I was having this issue with the original Microsoft implementation as well.

Sample app:

using System.Dynamic;

using Newtonsoft.Json.Converters;
using Newtonsoft.Json;

using RulesEngine.Models;

Workflow[] workflow = [
    new() {
        WorkflowName = "Workflow",
        Rules = [
            new() {
                RuleName = "TEST",
                Expression = "not things.Any(a == 1)",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            }
        ]
    }
];

List<string> payloads = [
    "{ \"things\": [] }",
    "{ \"things\": [ { \"a\": 1 } ] }",
    "{ \"things\": [ { \"a\": 2 } ] }"
];

var rulesEngine = new RulesEngine.RulesEngine(workflow, new() { IsExpressionCaseSensitive = false });

foreach(var payload in payloads) {
    dynamic expando = JsonConvert.DeserializeObject<ExpandoObject>(payload, new ExpandoObjectConverter())!;

    List<RuleResultTree> results = await rulesEngine.ExecuteAllRulesAsync("Workflow", expando);

    Console.WriteLine($"{payload}\n\t{results[0].IsSuccess} - {results[0].ExceptionMessage}");
}

output:

{ "things": [] }
        False - Exception while parsing expression `not things.Any(a == 1)` - The binary operator Equal is not defined for the types 'System.Object' and 'System.Int32'.
{ "things": [ { "a": 1 } ] }
        False -
{ "things": [ { "a": 2 } ] }
        True -

Any suggestions on how to solve this issue with empty arrays? I've gone as far as creating my own Iteration.Any custom type/method, but this makes the expressions excessively complicated.

Thank you!

[asulwer]

Any ArgumentNullException if source is null. this is expected behavior. your code is not reporting the Any exception, but it is the direct cause for the exception you see. i bet the inner exception would show the Any exception (or should)

[JasonBoggsAtHMSdotCom]

Source shouldn't be null though;, source should be an empty IEnumerable<object>. System.Linq.Any(source, predicate) should return false with empty arrays. The exception is being captured and handled within the rules engine.

I set IgnoreException and EnableExceptionAsErrorMessage to false and got this:

Thanks for your help!

[asulwer]

your issue is things.Any(a ==1), remove a==1 and now you are using System.Linq.Enumerable. The Any that you are using is a custom function?

Workflow[] workflow = [
    new() {
        WorkflowName = "Workflow",
        Rules = [
            new() {
                RuleName = "TEST",
                Expression = "not things.Any()",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            }
        ]
    }
];

List<string> payloads = [
    "{ \"things\": [] }",
    "{ \"things\": [ { \"a\": 1 } ] }",
    "{ \"things\": [ { \"a\": 2 } ] }" ];

var rulesEngine = new RulesEngine.RulesEngine(workflow, new() { IsExpressionCaseSensitive = false });

foreach (var payload in payloads)
{
    dynamic expando = JsonConvert.DeserializeObject<ExpandoObject>(payload, new ExpandoObjectConverter());

    List<RuleResultTree> results = await rulesEngine.ExecuteAllRulesAsync("Workflow", expando);

    Console.WriteLine($"{payload}\n\t{results[0].IsSuccess} - {results[0].ExceptionMessage}");
}

[JasonBoggsAtHMSdotCom]

No, it's not a custom function. The Any dynamic LINQ function can take a predicate to filter on the array's data https://dynamic-linq.net/expression-language#sequence-operators. I've tried using other dyanmic LINQ functions as well that take predicates, such as Where and All, and they all give the same error message.

[asulwer]

create an example that uses the Any method only. strip away RulesEngine.

[JasonBoggsAtHMSdotCom]

I figured it out. The issue was with how I was deserializing the payload.

This works:

...

JsonSerializerSettings settings = new();
settings.Converters.Add(new ExpandoObjectConverter());   

foreach(var payload in payloads) {
    var expando = JsonConvert.DeserializeObject(payload, settings)!;

    List<RuleResultTree> results = await rulesEngine.ExecuteAllRulesAsync("Workflow", expando);

    Console.WriteLine($"{payload}\n\t{results[0].IsSuccess} - {results[0].ExceptionMessage}");
}

Thanks again for all of your help.

[JasonBoggsAtHMSdotCom]

Unfortunately, I'm back. Using the solution I provided above causes issues with comparing integers. It looks like according to JSONDemo.cs, the correct method is to convert the string payloads to ExpandoObjects (using dynamic), and then sending these to the rules engine. when I do this, I receive additional errors. I'm using the following payload:

    {
        "prop": "value",
        "value": 3,
        "nest": {
            "code": "bar",
            "foo": true
        },
        "emptyArray": [],
        "populatedArray": [
            {
                "a": 2,
                "subArray": [
                    {
                        "c": 4
                    }
                ]
            }
        ]
    }

And these expressions: not emptyArray.Any(a == 1) returns error

Exception while parsing expression not emptyArray.Any(a == 1) - The binary operator Equal is not defined for the types 'System.Object' and 'System.Int32'.

populatedArray.Any(subArray.Any(c == 4)) returns error

Exception while parsing expression populatedArray.Any(subArray.Any(c == 4)) - The binary operator Equal is not defined for the types 'System.Object' and 'System.Int32'.

The following expressions do not throw errors: value > 1 prop = \"value\" nest.code eq \"bar\" and nest.foo == true

[asulwer]

@JasonBoggsAtHMSdotCom

the demo you followed is the original way to work with RulesEngine. the newer way or at least what i contributed is in the following example Json

can you modify your code to work using the example i gave?

[JasonBoggsAtHMSdotCom]

Here's an updated sample app

using System.Text.Json;
using System.Dynamic;

using RulesEngine.Models;

Workflow[] workflow = [
    new() {
        WorkflowName = "Workflow",
        Rules = [
            new() {
                RuleName = "value check",
                Expression = "value > 1",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            },
            new() {
                RuleName = "empty array",
                Expression = "not emptyArray.Any(a == 1)",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            },
            new() {
                RuleName = "populatedArray with subArray not match",
                Expression = "populatedArray.Any(subArray.Any(c == 4))",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            },
            new() {
                RuleName = "check prop",
                Expression = "prop = \"value\"",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            },
            new() {
                RuleName = "check nested code",
                Expression = "nest.code eq \"bar\" and nest.foo == true",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            }
        ]
    }
];

string payload = "{\"prop\":\"value\",\"value\":3,\"nest\":{\"code\":\"bar\",\"foo\":true},\"emptyArray\":[],\"populatedArray\":[{\"a\":2,\"subArray\":[{\"c\":4}]}]}";

var rulesEngine = new RulesEngine.RulesEngine(workflow, new() { IsExpressionCaseSensitive = false });

dynamic input1 = JsonSerializer.Deserialize<ExpandoObject>(payload)!;
var inputs = new[] { input1 };

CancellationTokenSource cancellationTokenSource = new();
List<RuleResultTree> results = await rulesEngine.ExecuteAllRulesAsync("Workflow", cancellationTokenSource.Token, inputs);

Console.WriteLine(payload);
Console.WriteLine();

foreach(var result in results) {
    Console.WriteLine($"\t{result.IsSuccess} - {result.ExceptionMessage}");
}

this outputs:

{"prop":"value","value":3,"nest":{"code":"bar","foo":true},"emptyArray":[],"populatedArray":[{"a":2,"subArray":[{"c":4}]}]}

        True -
        False - Exception while parsing expression `not emptyArray.Any(a == 1)` - The binary operator Equal is not defined for the types 'System.Object' and 'System.Int32'.
        False - Exception while parsing expression `populatedArray.Any(subArray.Any(c == 4))` - The binary operator Equal is not defined for the types 'System.Object' and 'System.Int32'.
        True -
        True -

I will review the Json.cs example. Due to the way the data is passed in, I'm not able to use RuleParameter.

[JasonBoggsAtHMSdotCom]

Just for completeness, I tried using RuleParameter, even though it wouldn't work with how our rules are designed, and it yields the same results

using System.Text.Json;
using System.Dynamic;

using RulesEngine.Models;

Workflow[] workflow = [
    new() {
        WorkflowName = "Workflow",
        Rules = [
            new() {
                RuleName = "value check",
                Expression = "input1.value > 1",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            },
            new() {
                RuleName = "empty array",
                Expression = "not input1.things.Any(a == 1)",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            },
            new() {
                RuleName = "populatedArray with subArray not match",
                Expression = "input1.populatedArray.Any(subArray.Any(c == 4))",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            },
            new() {
                RuleName = "check prop",
                Expression = "input1.prop = \"value\"",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            },
            new() {
                RuleName = "check nested code",
                Expression = "input1.nest.code eq \"bar\" and input1.nest.foo == true",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            }
        ]
    }
];

string payload = "{\"prop\":\"value\",\"value\":3,\"nest\":{\"code\":\"bar\",\"foo\":true},\"emptyArray\":[],\"populatedArray\":[{\"a\":2,\"subArray\":[{\"c\":4}]}]}";
dynamic input1 = JsonSerializer.Deserialize<ExpandoObject>(payload)!;

var rp = new RuleParameter[] {
    new("input1", input1)
};

var rulesEngine = new RulesEngine.RulesEngine(workflow, new() { IsExpressionCaseSensitive = false });

var inputs = new[] { input1 };

CancellationTokenSource cancellationTokenSource = new();
List<RuleResultTree> results = await rulesEngine.ExecuteAllRulesAsync("Workflow", cancellationTokenSource.Token, inputs);

Console.WriteLine(payload);
Console.WriteLine();

foreach(var result in results) {
    Console.WriteLine($"\t{result.IsSuccess} - {result.ExceptionMessage}");
}

{"prop":"value","value":3,"nest":{"code":"bar","foo":true},"emptyArray":[],"populatedArray":[{"a":2,"subArray":[{"c":4}]}]}

        True -
        False - Exception while parsing expression `not input1.things.Any(a == 1)` - The binary operator Equal is not defined for the types 'System.Object' and 'System.Int32'.
        False - Exception while parsing expression `input1.populatedArray.Any(subArray.Any(c == 4))` - The binary operator Equal is not defined for the types 'System.Object' and 'System.Int32'.
        True -
        True -

[asulwer]

@JasonBoggsAtHMSdotCom

i changed your code a bit. in your previous issue you discovered a solution, which works for this one

JsonSerializerSettings settings = new();
settings.Converters.Add(new ExpandoObjectConverter());

dynamic input1 = JsonConvert.DeserializeObject(payload, settings)!;

[asulwer]

@JasonBoggsAtHMSdotCom

both examples work as expected

There are two method signatures for ExecuteAllRulesAsync (with CancellationToken as a parameter), original and newish

Original

List<RuleResultTree> results = await rulesEngine.ExecuteAllRulesAsync("Workflow", cancellationToken, inputs);

var inputs = new[] {
    new {
        prop = "value",
        value = 3,
        emptyArray = Array.Empty<object>(),
        populatedArray = new[] {
            new {
                a = 2,
                subArray = new[] {
                    new {
                        c = 4
                    }
                }
            }
        },
        nest = new {
            code = "bar",
            foo = true
        }
    }
};

var rulesEngine = new RulesEngine.RulesEngine(workflows);

List<RuleResultTree> results = await rulesEngine.ExecuteAllRulesAsync("Workflow", cancellationToken, inputs);

Alternate - preferred

List<RuleResultTree> results = await rulesEngine.ExecuteAllRulesAsync("Workflow", rp, cancellationToken);

var ruleParameters = new RuleParameter[] {
    new("prop", "value"),
    new("value", 3),
    new("emptyArray", Array.Empty<object>()),
    new("populatedArray", new[] {
        new {
            a = 2,
            subArray = new[] {
                new {
                    c = 4
                }
            }
        }
    }),
    new("nest", new {
        code = "bar",
        foo = true
    })
};

var rulesEngine = new RulesEngine.RulesEngine(workflows);

List<RuleResultTree> results = await rulesEngine.ExecuteAllRulesAsync("Workflow", ruleParameters, cancellationToken);

[JasonBoggsAtHMSdotCom]

I am not able to send .NET objects - my project must start from a json string. I tried using dynamic input1 = JsonConvert.DeserializeObject(payload, settings)!; as suggested and it still results in an issue on the value > 1 comparison. I'm going to try breaking this down to see if the issue is happening in LINQ or the rules engine. I also have some other ideas to try today. I greatly appreciate your guidance.

[asulwer]

it worked with the json but converted using your original solution

JsonSerializerSettings settings = new(); settings.Converters.Add(new ExpandoObjectConverter());

foreach(var payload in payloads) { var expando = JsonConvert.DeserializeObject(payload, settings)!;

List<RuleResultTree> results = await rulesEngine.ExecuteAllRulesAsync("Workflow", expando);

Console.WriteLine($"{payload}\n\t{results[0].IsSuccess} - {results[0].ExceptionMessage}");

}

[JasonBoggsAtHMSdotCom]

I ultimately was not able to get this working with deserializing a json payload to an anonymous type (as an expando or otherwise) and use the Any(expression) functions against it. I traced this deep into System.Linq.Dynamic.Core and do not believe this to actually be a problem with RulesEngine. I'm looking at other options to strongly type my objects, by either moving the RulesEngine directly into my integration, or by importing assemblies and referencing the types by namespace and class names.

[asulwer]

to be clear about your requirements, incoming json deserialized to an expandoobject, mandatory? if those are your requirements then i can tailor my examples or modify code accordingly.

if you think the issue is with System.Linq.Dynamic.Core, which you wouldn't be the first, i can open a ticket over there, on your behalf? lets work together to solve this, i enjoy the challenge

[asulwer]

@JasonBoggsAtHMSdotCom according to a few minutes ago, there is a new version for System.Linq.Dynamic.Core 1.4.8. RulesEngine targets, in version 6.0.5, i think, 1.4.6

All of your example code, so far, i have been able to get working in one form or fashion. can you provide some examples that i can duplicate your issue but cannot 'fix'? i can test before the upgrade and then after, maybe they 'fixed' their code? they do not publish release notes, that i can find.

also, where specifically were you able to narrow the error down to?

[asulwer]

@JasonBoggsAtHMSdotCom System.Linq.Dynamic.Core has two examples on how to use their Any method, both use EF context. E1 and E2

i have combined both examples into one. Can you modify them to recreate your issue? LinqDynamicAny-Example.zip

[asulwer]

@JasonBoggsAtHMSdotCom we recently converted from Newtonsoft.Json to System.Text.Json, which in my opinion is not helping with your issue.

The older way of using this code expects as input an ExpandoObject. System.Text.Json child elements are kept as JsonElement's.

try this code to completely convert to ToExpandObject and maybe this one too GetTypedObject?

[JasonBoggsAtHMSdotCom]

to be clear about your requirements, incoming json deserialized to an expandoobject, mandatory? if those are your requirements then i can tailor my examples or modify code accordingly.

if you think the issue is with System.Linq.Dynamic.Core, which you wouldn't be the first, i can open a ticket over there, on your behalf? lets work together to solve this, i enjoy the challenge

For now, yes. The requirements I have are to create a generic tool that can parse any payload using a simple expression. I pulled down RulesEngine and stepped through it. On line 40 (return new ExpressionParser(parameters, expression, new object[] { }, config).Parse(returnType);) of src\RulesEngine\ExpressionBuilders\RuleExpressionParser.cs, it hits an exception when it enters the Parse method. The exception is the Exception while parsing expression... error mentioned earlier.

I also tried deserializing without an explicit type using Newtonsoft.Json, which actually converts it to a JObject, and the Any expressions work there, but this method has issues with comparing against integers - I'm forced to cast integers using int(. Ideally, I would not use newtonsoft at all and I'd like to get this working using an ExpandoObject.

I'll work on creating a simplified end-to-end example.

Thanks for taking the time to work with me through this.

[JasonBoggsAtHMSdotCom]

In the code below, I have 3 options for Deserializing the payload (string) before sending it to the Rules engine with 5 rules. All 3 options experience an exception of some sort. In a separate prototype, I created an object that represented the data model in my example and Deserialized to that typed object, and all expressions worked correctly. Unfortunately strongly typed objects are not easily attainable for my requirements.

using System.Collections;
using System.Dynamic;
using System.Text.Json;

using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

using RulesEngine.HelperFunctions;
using RulesEngine.Models;

string payload = "{\"prop\":\"someString\",\"someInt\":3,\"nest\":{\"code\":\"bar\",\"foo\":true},\"emptyArray\":[],\"populatedArray\":[{\"a\":2,\"subArray\":[{\"c\":4}]}]}";

Workflow[] workflow = [
    new() {
        WorkflowName = "Workflow",
        Rules = [
            new() {
                RuleName = "someInt check",
                Expression = "someInt > 1",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            },
            new() {
                RuleName = "empty array",
                Expression = "not emptyArray.Any(a == 3)",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            },
            new() {
                RuleName = "populatedArray with subArray not match",
                Expression = "populatedArray.Any(subArray.Any(c == 4))",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            },
            new() {
                RuleName = "check prop",
                Expression = "prop = \"someString\"",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            },
            new() {
                RuleName = "check nested code",
                Expression = "nest.code eq \"bar\" and nest.foo == true",
                RuleExpressionType = RuleExpressionType.LambdaExpression
            }
        ]
    }
];

var rulesEngine = new RulesEngine.RulesEngine(workflow, new() {
    IsExpressionCaseSensitive = false,
    CustomTypes = new[] {
        typeof(IEnumerable)
    }
});

var options = new JsonSerializerOptions();

//// ------ Replace between these lines ------

// Option 1 - Deserialize to ExpandoObject
var target = System.Text.Json.JsonSerializer.Deserialize<ExpandoObject>(payload, options)!;

// Option 2 - Deserialize to JsonElement then ExpandoObject via ToExpandoObject
//var target = System.Text.Json.JsonSerializer.Deserialize<JsonElement>(payload, options)!.ToExpandoObject();

// Option 3 - Deserialize to JObject using Newtonsoft
//var target = Newtonsoft.Json.JsonConvert.DeserializeObject(payload, new JsonSerializerSettings())!;

//// ------ Replace between these lines ------

CancellationTokenSource cancellationTokenSource = new();
List<RuleResultTree> results = await rulesEngine.ExecuteAllRulesAsync("Workflow", cancellationTokenSource.Token, target);

if(target.GetType() == typeof(JObject)) {
    var ns = Convert.ChangeType(target, typeof(JObject));
    Console.WriteLine(Newtonsoft.Json.JsonConvert.SerializeObject(ns, new JsonSerializerSettings { Formatting = Formatting.Indented}));
}
else {
    Console.WriteLine(System.Text.Json.JsonSerializer.Serialize(target, new JsonSerializerOptions { WriteIndented = true}));
}
Console.WriteLine();

foreach(var result in results) {
    Console.WriteLine($"\t{result.IsSuccess}\t{result.Rule.Expression}");
    if(result.ExceptionMessage.Length > 0) Console.WriteLine($"\t\t\t{result.ExceptionMessage}");
}

[asulwer]

thanks for the code location. i will start there and see if i can solve this on my end or create a issue ticket

[asulwer]

@JasonBoggsAtHMSdotCom the issue is with either of the two Deserialize methods used. each has a tick.

this helped, its maybe a start? the only issue with it is the empty array, so maybe that code can be changed to handle that?

[JasonBoggsAtHMSdotCom]

Thanks for providing that link. I tried the custom converter and it didn't seem to have any impact on the result.

I've been digging in to this a little bit, and I believe the issue is in Utils.cs we iterate through the expando object and create anonymous typed objects from the first value in the expando. The issue is when we get to the empty array, it doesn't know what the underlying type is, so the generic type argument is an object (Line 54 value = typeof(List<object>);). It needs to be an anonymous type, but at this point we don't know what fields need to be there, so there's no way to create a dummy record and Take(0). If I create an anonymous type with an object of a, it actually succeeds.

I'm still a little lost on why Any( even attempts to process the underlying records when the list itself is empty - it can never be true on an empty list.

[asulwer]

when we iterate through the properties it is not being converted correctly, which is why each of your examples resulted in different issues. Utils is a good place for me to start.

again, thanks for all of your help narrowing this issue down.

[JasonBoggsAtHMSdotCom]

I broke this down to exclude the rules engine entirely and I can definitely say this is an issue with system.linq, and maybe this is by design (albeit, ~a bad~ an unfavorable one).

using System.Linq.Expressions;
using System.Linq.Dynamic.Core;

string payload = "{\"prop\":\"someString\",\"someInt\":3,\"nest\":{\"code\":\"bar\",\"foo\":true},\"emptyArray\":[],\"populatedArray\":[{\"a\":2,\"subArray\":[{\"c\":4}]}]}";

List<string> expressions = [
    "someInt > 1",
    "not emptyArray.Any(a == 3)",
    "prop = \"someString\""
];

var genericList = typeof(List<>).MakeGenericType(new {  }.GetType());
//var genericList = typeof(List<>).MakeGenericType(new { a = 300 }.GetType());

var _props = new List<DynamicProperty> {
        new("someInt", typeof(int)),
        new("prop", typeof(string)),
        new("emptyArray", genericList)
    };

Type anonType = DynamicClassFactory.CreateType(_props);

var target = System.Text.Json.JsonSerializer.Deserialize(payload, anonType);

foreach(var expression in expressions) {
    LambdaExpression? lambda;
    try {
        lambda = DynamicExpressionParser.ParseLambda(anonType, typeof(bool), expression);
    }
    catch {
        Console.WriteLine($"Error with: {expression}");
        continue;
    }
    var result = lambda?.Compile().DynamicInvoke(target);

    Console.WriteLine(result?.ToString() + ": " + expression);
}

If you swap the lines that create genericList, the line where it creates a generic object with an a property works. When evaluating an array object with a predicate, System.Linq simply does not check for the presence of any records before evaluating the contents of object. Ideally Any would check simply return false if the Count/Length was zero. This does get tricky though with Where, as it would need to know what type of object to return, even if the list/array is empty.

[asulwer]

what would be the harm if the array is empty adding something to it as the default?

[JasonBoggsAtHMSdotCom]

Nothing, but I have no way of knowing what that object will look like. The application I'm working with has roughly 750 models and the property could literally be anything. if I put something else in there as a dummy property, the rule still blows up.

using System.Linq.Expressions;
using System.Linq.Dynamic.Core;

string payload = "{\"prop\":\"someString\",\"someInt\":3,\"nest\":{\"code\":\"bar\",\"foo\":true},\"emptyArray\":[],\"populatedArray\":[{\"a\":2,\"subArray\":[{\"c\":4}]}]}";

List<string> expressions = [
    "someInt > 1",
    "not emptyArray.Any(a == 3)",
    "prop = \"someString\""
];

//var genericList = typeof(List<>).MakeGenericType(new {  }.GetType());
//var genericList = typeof(List<>).MakeGenericType(new { a = 300 }.GetType());
var genericList = typeof(List<>).MakeGenericType(new { dummy = true }.GetType());

var _props = new List<DynamicProperty> {
        new("someInt", typeof(int)),
        new("prop", typeof(string)),
        new("emptyArray", genericList)
    };

Type anonType = DynamicClassFactory.CreateType(_props);

var target = System.Text.Json.JsonSerializer.Deserialize(payload, anonType);

foreach(var expression in expressions) {
    LambdaExpression? lambda;
    try {
        lambda = DynamicExpressionParser.ParseLambda(anonType, typeof(bool), expression);
    }
    catch(Exception ex) {
        Console.WriteLine($"Error with: {expression}: {ex.Message}");
        continue;
    }
    var result = lambda?.Compile().DynamicInvoke(target);

    Console.WriteLine(result?.ToString() + ": " + expression);
}

The second expression results in an error: Error with: not emptyArray.Any(a == 3): No property or field 'a' exists in type '<>f__AnonymousType0`1'

[asulwer]

Note: i didn't try this with option3 Newtonsoft, only the first two options, as you can see in the example

i have a working example for you. the empty array is still an issue, maybe its System.Linq.Dynamic.Core, still investigating. regardless, the issue only exists with the empty array and it can be caught, the two options have different errors for the empty array. I had to drastically change Utils - original & Utils - altered.

@RenanCarlosPereira Before i merge the Empty Array branch, i will need the original author for that code to review my 'hack'

Empty Array - branch

refer to these two files for the included changes Example & Utils

[JasonBoggsAtHMSdotCom]

I think I might have figured this out; I created a new class (called ImplicitObject) that has a bunch of implicit operators, and then instead of creating a List<object> I create a list of a dictionary of this class (List<Dictionary<string, ImplicitObject>>). This does make the assumption that list items will take the form of a dictionary, but it will only happen when a list is empty, so I don't think it'll cause any issues. I'll submit a PR later today.

[JasonBoggsAtHMSdotCom]

https://github.com/asulwer/RulesEngine/pull/79

[asulwer]

@JasonBoggsAtHMSdotCom of the three options you provided i assume this only works with ExpandoObject's? i did not try my solution against your Newtonsoft option. i am concerned your solution doesnt work with the JsonElement option. i really cannot see anyone casting that way. usually ExpandoObject or RulesParameter.

just my thoughts before i commit

[JasonBoggsAtHMSdotCom]

Correct - my solution only addresses Expando Objects. The example I provided in #79 uses newtonsoft and the ExpandoObjectConveter class to convert to an expando, but I will be using System.Text.Json and a custom conveter in my own solution to convert the json payload to an expando, see the accepted solution here https://stackoverflow.com/questions/65972825/c-sharp-deserializing-nested-json-to-nested-dictionarystring-object

[asulwer]

this was closed when i merged your code. the PR only works with Newtonsoft and not System.Text.Json

[asulwer]

i am going to integrate the stackoverflow solution as a possible for System.Text.Json options you provided.

[RenanCarlosPereira]

hey, didnt have time to check the code yet... I will take a look in the thread first

[asulwer]

@RenanCarlosPereira it is possible that @JasonBoggsAtHMSdotCom created a solution. i merged his code which works only with Newtonsoft deserializing an ExpandoObject. the link he provided to StackExchange works with System.Text.Json. i added an example with that code so show it working

[RenanCarlosPereira]

did we created unit test to reproduce the issue?

[asulwer]

i am working with another branch regarding this issue

[RenanCarlosPereira]

I dont think we have an issue here, let me add a sample, so you guys can check

[JasonBoggsAtHMSdotCom]

@RenanCarlosPereira it is possible that @JasonBoggsAtHMSdotCom created a solution. i merged his code which works only with Newtonsoft deserializing an ExpandoObject. the link he provided to StackExchange works with System.Text.Json. i added an example with that code so show it working

Ultimately the issue is that the System.Text.Json deserializer doesn't properly deserialize json payloads to an ExpandoObject the way one would expect (which is the way newtonsoft does when you use their expando object converter). the system.text.json deserializer converts everything to a JsonElement.

Consider the following:

using System.Dynamic;

string payload = "{\"prop\":\"someString\",\"someInt\":3,\"nest\":{\"code\":\"bar\",\"foo\":true},\"emptyArray\":[],\"populatedArray\":[{\"a\":2,\"subArray\":[{\"c\":4}]}]}";

var stj = System.Text.Json.JsonSerializer.Deserialize<ExpandoObject>(payload);
var nsj = Newtonsoft.Json.JsonConvert.DeserializeObject<ExpandoObject>(payload, new Newtonsoft.Json.Converters.ExpandoObjectConverter());

var stj_d = (IDictionary<string, object>)stj!;
var nsj_d = (IDictionary<string, object>)nsj!;

Console.WriteLine(stj_d["prop"].GetType());
Console.WriteLine(nsj_d["prop"].GetType());

In this, System.Text.Json converted the "prop" property to a System.Text.Json.JsonElement, whereas newtonsoft properly converts it to a System.String. The system.text.json expando is full of JsonElements

And newtonsoft is properly typed

Using the conversion in the stack overflow link, system text json properly converts the same way newtonsoft does. I renamed the name of the converter from the stack overflow link to ExpandoObjectConverter

[asulwer]

@JasonBoggsAtHMSdotCom i added the stackexchange solution you found as an example in the demo project with references to this issue and stackexchange. my currently opoen PR should resolve this issue

[RenanCarlosPereira]

In Microsoft's Rules Engine, C# expressions are dynamically evaluated, which can introduce issues with type inference, especially when working with arrays. Here’s a breakdown of the problem:

1. Dynamic Typing in Expressions: When an empty array (Array.Empty<dynamic>()) is passed, the compiler does not infer a specific type for the elements in the array. Consequently, properties within the array items (like a) are typed as object.

2. Type Mismatch: In C#, comparisons between int and object are not allowed directly. Therefore, when the Rules Engine tries to execute an expression like things.Any(x => x.a == 1), it fails because x.a is typed as object, and it cannot be compared directly to 1 (an int).

3. Why Sample Works but Rules Engine Fails: When you create a sample array directly in C# (e.g., new[] { new { a = 1 }, new { a = 2 } }), the compiler can deduce the type for a, which is why sample.things.Any(x => x.a == 1) works. However, in the Rules Engine, which processes expressions dynamically at runtime, this type inference does not happen, leaving a as object.

using System;
using System.Collections.Generic;
using System.Dynamic;
using System.Linq;
using System.Text.Json;
using Newtonsoft.Json;
using Newtonsoft.Json.Converters;
using RulesEngine.Models;

public class Program
{
    public static async Task Main()
    {
        // Define a workflow with a rule that checks if any item in 'things' has an 'a' property equal to 1
        Workflow[] workflows = new[]
        {
            new Workflow
            {
                WorkflowName = "Workflow",
                Rules = new List<Rule>
                {
                    new Rule
                    {
                        RuleName = "TEST",
                        Expression = "things.Any(x => x.a == 1) == True",
                        RuleExpressionType = RuleExpressionType.LambdaExpression
                    }
                }
            }
        };

        // Define payloads with different 'things' arrays
        List<object> payloads = new List<object>
        {
            // Case 1: Empty array - causes issue because the type of 'a' is undetermined
            new { things = Array.Empty<dynamic>() },

            // Case 2: Array with items - type of 'a' inferred as int, so comparison works
            new { things = new[] { new { a = 1 }, new { a = 2 } } },

            // Case 3: Array with items - type of 'a' inferred as int, comparison still works
            new { things = new[] { new { a = 3 }, new { a = 4 } } }
        };

        // Test the expression directly in C# (to show it works with static typing)
        var sample = new { things = new[] { new { a = 1 }, new { a = 2 } } };
        bool directResult = sample.things.Any(x => x.a == 1);
        Console.WriteLine($"Direct C# check: {directResult}"); // Outputs: True

        // Instantiate the Rules Engine
        var rulesEngine = new RulesEngine.RulesEngine(workflows);

        // Process each payload in the Rules Engine
        foreach (var payload in payloads)
        {
            // Serialize payload to JSON and then deserialize to ExpandoObject for dynamic typing
            var jsonPayload = JsonConvert.SerializeObject(payload);
            var expando = JsonConvert.DeserializeObject<ExpandoObject>(jsonPayload, new ExpandoObjectConverter())!;

            // Convert ExpandoObject properties to RuleParameters for Rules Engine
            var ruleParameters = expando.Select(x => new RuleParameter(x.Key, x.Value)).ToArray();

            // Execute rules and capture results
            List<RuleResultTree> results = await rulesEngine.ExecuteAllRulesAsync("Workflow", ruleParameters);

            // Display results for each payload
            Console.WriteLine($"Payload: {jsonPayload}");
            Console.WriteLine($"Rule Success: {results[0].IsSuccess}");
            Console.WriteLine($"Exception Message: {results[0].ExceptionMessage}");
            Console.WriteLine();
        }
    }
}

[RenanCarlosPereira]

about this https://github.com/asulwer/RulesEngine/issues/75?notification_referrer_id=NT_kwDOANVbW7QxMjk3NDA5Mzg5NjoxMzk4MjU1NQ#issuecomment-2474502348

@JasonBoggsAtHMSdotCom you are 100% right, System.Text.Json does the serialization using JsonElement, thats a pain in the ass, if we dont know the type its better to use the Newtonsoft.Json.

in other hand, the empty array is a trick challange, as not even c# could inver the array type upfront.

[JasonBoggsAtHMSdotCom]

Your summary seems accurate - the Utils class was defaulting empty lists to List<object>, and expressions run against these empty arrays, such as things.Any(a == 3) fail in the parser engine because object doesn't have any addressable properties. Changing List<object> to List<Dictionary<string,ImplicitObject>>, along with all of the implicit operators in the ImplicitObject class, allows the expression engine to realize the underlying object can be converted to just about any valid json type (bool, char, string, numerics, etc).

@asulwer will we be able to push this change to NuGet soon?

[RenanCarlosPereira]

Before publishing a nuget, could we implement a unit test covering this situation? so if somebody in the future decides to change those ImplicitObject the test might catch it.

[JasonBoggsAtHMSdotCom]

unit tests added https://github.com/asulwer/RulesEngine/pull/83

[RenanCarlosPereira]

The build didn't pass, could you check?

[JasonBoggsAtHMSdotCom]

https://github.com/asulwer/RulesEngine/pull/84