fail-if warn-if fail with large arrays

[oliver-sanders]

The any and all functions supported by the rose mini-language expand expressions in full e.g. any(foo == 1) would be evaluated as:

if foo_1 == 1 and foo_2 == 1 and foo_3 == 1 ... foo_n == 1

Backstage jinja2 is used to evaluate this expression but it would appear jinja2 falls over raising MemoryError for large arrays (confirmed with 165 elements).

[oliver-sanders]

Jinja2 was used here for security reasons, however, Jinja2 is a fairly powerful programming language so not as locked down as we might want it to be.

A better solution is probably to do it in Python, here is an example in ast (other syntax trees are available):

class SimpleVisitor(ast.NodeVisitor):
    """Abstract syntax tree node visitor for simple safe operations."""

    def visit(self, node):
        if not isinstance(node, self.whitelist):
            # permit only whitelisted operations
            raise ValueError(type(node))
        return super().visit(node)

    whitelist = (
        ast.Expression,
        # variables
        ast.Name, ast.Load, ast.Attribute, ast.Subscript, ast.Index,
        # opers
        ast.BinOp, ast.operator,
        # types
        ast.Num, ast.Str,
        # comparisons
        ast.Compare, ast.cmpop, ast.List, ast.Tuple
    )

def simple_eval(expr, **variables):
    """Safely evaluates simple python expressions.

    Supports a minimal subset of Python operators:
    * Binary operations
    * Simple comparisons

    Supports a minimal subset of Python data types:
    * Numbers
    * Strings
    * Tuples
    * Lists

    Examples:
        >>> simple_eval('1 + 1')
        2
        >>> simple_eval('1 < a', a=2)
        True
        >>> simple_eval('1 in (1, 2, 3)')
        True
        >>> import psutil
        >>> simple_eval('a.available > 0', a=psutil.virtual_memory())
        True

        If you try to get it to do something it's not supposed to:
        >>> simple_eval('open("foo")')
        Traceback (most recent call last):
        ValueError: open("foo")

    """
    try:
        node = ast.parse(expr.strip(), mode='eval')
        SimpleVisitor().visit(node)
        return eval(
            compile(node, '<string>', 'eval'),
            {'__builtins__': None},
            variables
        )
    except Exception:
        raise ValueError(expr)

[oliver-sanders]

Bumping this to 2.x.

Need to confirm where the limit is for lastest Python / Jinja2 versions.

The real solution is to move from Jinja2 to Python for this functionality using come careful AST logic.