Rule blocked: CORERULES-9695 - validation now fails when using empty/non_empty with missing variable

A .txt file with the JSON containing the full request: Request.txt

Links to related JIRA Tickets

Rule Information

Standard: USDM
Rule ID: DDF00152
Rule Description: An activity must only reference timelines that are specified within the same study design.

Describe the bug Since the implementation of advanced error handling (main branch commit c5273ea1774c83cafe7d3ba6224cba1e283c6a26), execution of any validation containing the empty and non_empty operators fails with several errors when the checked variable does not exist in the dataset. Previously, the resultant KeyError exception would have halted validation and been picked up by the handle_validation_exceptions method. The absence of the variable would just be reported as "Column not found in data", which would be interpreted and reported by the Rules Editor as one of the "Absent-Variable Skips". Now, the error is reported via the terminal (with a traceback), but validation does not halt, so subsequent errors occur due to the missing result for empty/non_empty. The last error is then returned and reported by the Rules Editor.

This bug is likely to affect a LOT of rules

Error returned from Rule Engine This message is the result of the last exception, when the None result of the failed empty/non_empty operator is merged with results from other operators.

{
    ...,
    "SCHEDULETIMELINE": [
        {
            "executionStatus": "execution_error",
            "dataset": "ScheduleTimeline.xpt",
            "domain": "SCHEDULETIMELINE",
            "variables": [],
            "message": "rule execution error",
            "errors": [
                {
                    "dataset": "ScheduleTimeline.xpt",
                    "error": "An unknown exception has occurred",
                    "message": "Cannot perform 'and_' with a dtyped [bool] array and scalar of type [NoneType]"
                }
            ]
        }
    ]
}

Expected behavior As described in the Boolean Logic section of the Reference, the empty and non_empty operators should result in an absent variable skip when the checked variable does not exist in the dataset.

cdisc-org / cdisc-rules-engine

Rule blocked: CORERULES-9695 - validation now fails when using empty/non_empty with missing variable #932