How to allow a package completely?

[wildsonc]

BUG/PROBLEM REPORT / FEATURE REQUEST

What I did:

Added package to safe_globals:

import dateutil

safe_globals = {
# Others
"dateutil": dateutil
}

Code

dateutil.parser.parse("2024-01-01").strftime("%d/%m/%Y")

What I expect to happen:

Result: "01/01/2024"

What actually happened:

Error: '__import__'

This code code works:

dateutil.parser.parse("2024-01-01").date()

If the method has any import, it is not allowed

[d-maurer]

Wil wrote at 2024-6-28 13:12 -0700:

... Error: '__import__'

This code code works:

dateutil.parser.parse("2024-01-01").date()

If the method has any import, it is not allowed

This is surprising: If you make a module available to restricted code, then access to the its attributes (recursively) inside the restricted code is supervised by RestrictedPython. But any activity inside the module itself is not restricted. Especially, it should not matter whether the module imports anything.

Your problem description is too terse. Please provide a complete piece of code which reproduces the problem. I.e. your code should show

• your additions to safe_globals
• which restricted source you compile in what way
• how you execute the restricted code

[wildsonc]

import json
import re
from datetime import datetime, time, timedelta, timezone
from typing import Any

import dateutil
import dateutil.parser
import requests
from core.plugins.salesforce import Salesforce
from RestrictedPython import (
    Eval,
    Guards,
    compile_restricted_function,
    limited_builtins,
    safe_builtins,
    utility_builtins,
)

from .utils import CompileError

allowed_imports = {
    "datetime": datetime,
    "dateutil": dateutil,
    "enumerate": enumerate,
    "format_str": lambda x, y: x.format(y),
    "strftime": lambda x, y: x.strftime(y),
    "json": json,
    "dict": dict,
    "min": min,
    "max": max,
    "all": all,
    "any": any,
    "sum": sum,
    "re": re,
    "requests": requests,
    "Salesforce": Salesforce,
    "time": time,
    "timedelta": timedelta,
    "timezone": timezone,
}

ALLOWED_BUILTINS = {}
ALLOWED_BUILTINS.update(safe_builtins)
ALLOWED_BUILTINS.update(limited_builtins)
ALLOWED_BUILTINS.update(utility_builtins)

safe_globals: dict[str, Any] = dict(
    __builtins__=ALLOWED_BUILTINS,
    _write_=Guards.full_write_guard,
    _getiter_=Eval.default_guarded_getiter,
    _iter_unpack_sequence_=Guards.guarded_iter_unpack_sequence,
    getattr=Guards.safer_getattr,
    setattr=Guards.guarded_setattr,
    delattr=Guards.guarded_delattr,
    **allowed_imports,
)

class Script:
    def __init__(self, *args, **kwargs) -> None:
        pass

    def run(self, code: str, parameters: dict = {}):
        if not code:
            return None

        function_name = "script"
        function_parameters = ", ".join(parameters.keys())
        compiled = compile_restricted_function(
            function_parameters,
            code,
            function_name,
            filename="<inline code>",
        )
        if compiled.errors:
            raise CompileError(compiled.errors)

        safe_locals = dict()

        exec(compiled.code, safe_globals, safe_locals)
        function = safe_locals[function_name]

        result = function(**parameters)
        return result

[d-maurer]

Wil wrote at 2024-6-28 13:58 -0700:

... The reason why I asked for complete code is that I would like to run your code (in an environment with RestrictedPYthon) to reproduce the problem: i.e. "python should reproduce the problem. This means, your code should only reference standard modules and packages (and e.g. notcore.plugins.salesform`).

You have in the latest message provided some infrastructure -- but the concrete problematic code is not contained. Keep in mind: your code should reproduce the problem when run in an environment with RestrictedPython (and otherwise only standard packages).

[wildsonc]

Hey @d-maurer, try this version now:

import dateutil
from RestrictedPython import (
    Eval,
    Guards,
    compile_restricted,
    safe_builtins,
)

allowed_imports = {
    "dateutil": dateutil,
}

safe_global = dict(
    __builtins__=safe_builtins,
    _write_=Guards.full_write_guard,
    _getiter_=Eval.default_guarded_getiter,
    _iter_unpack_sequence_=Guards.guarded_iter_unpack_sequence,
    getattr=Guards.safer_getattr,
    setattr=Guards.guarded_setattr,
    delattr=Guards.guarded_delattr,
    **allowed_imports,
)

source_code = """
dt = dateutil.parser.parse("2024-01-01").strftime("%d/%m/%Y")
"""

# Working code
# source_code = """
# dt = dateutil.parser.parse("2024-01-01").date()
# """

compiled = compile_restricted(source_code)

safe_locals = dict()
exec(compiled, safe_global, safe_locals)
print(safe_locals["dt"])

[d-maurer]

Wil wrote at 2024-7-1 12:47 -0700:

Hey @d-maurer, try this version now:

import dateutil ...

Almost (but not yet completely) there: apparently, dateutil does not belong to the standard Python library. I looked on PyPI but apparently there are several candidates. Which dateutil distribution do you use?

[wildsonc]

Sorry for the delay, I use this package:

https://github.com/dateutil/dateutil

An example using only datetime instead dateutil:

from datetime import datetime

from RestrictedPython import Eval, Guards, compile_restricted, safe_builtins

allowed_imports = {"datetime": datetime}

safe_global = dict(
    __builtins__=safe_builtins,
    _write_=Guards.full_write_guard,
    _getiter_=Eval.default_guarded_getiter,
    _iter_unpack_sequence_=Guards.guarded_iter_unpack_sequence,
    getattr=Guards.safer_getattr,
    setattr=Guards.guarded_setattr,
    delattr=Guards.guarded_delattr,
    **allowed_imports,
)

source_code = """
dt = datetime.now().strftime("%d/%m/%Y")
"""

compiled = compile_restricted(source_code)

safe_locals = dict()
exec(compiled, safe_global, safe_locals)
print(safe_locals["dt"])

[d-maurer]

Wil wrote at 2024-7-16 08:20 -0700:

... An example using only datetime instead dateutil: ...

A minimal example reproducing the problem is

from datetime import datetime
from RestrictedPython import compile_restricted, safe_builtins

safe_globals = dict(__builtins__=safe_builtins,
                    strftime=datetime.now().strftime)

code = compile_restricted("strftime('%Y')\n")
exec(code, safe_globals)

The reason: strftime is implemented in "C", i.e. not a "normal" Python function. A normal Python function accesses globals (and among them builtins) via its __globals__ attribute; functions implemented in "C" lack this possibility. The implementation of strftime tries to import Python's time module via a call to __import__. Because it lacks its own __globals__ it tries to find __import__ in the calling frame -- which happens in your case to be restricted and lacks __import__.

RestrictedPython cannot do anything for this special situation.

I see the following possibilities to work around the problem:

• put an __import__ definition in safe_builtins which allows to import the time module (and maybe reject any other import).

• wrap the "C" function strftime into a normal Python function (defined in an unrestricted context).

[wildsonc]

Could you show me an example of how to implement import? Allowing to import only the time module

[d-maurer]

Wil wrote at 2024-7-22 05:22 -0700:

Could you show me an example of how to implement import? Allowing to import only the time module

No, I am not ready to do your work.

But it is not difficult: you read the Python documentation how __import__ is called and what it is expected to do. You do what is expected of __import__ for the modules you would like to be importable (at least "time" in your case).

[wildsonc]

This way it worked as expected:

from RestrictedPython import compile_restricted, safe_builtins

allowed_imports = ["datetime", "time", "dateutil"]

def guarded_import(name, globals=None, locals=None, fromlist=(), level=0):
    base_name = name.split(".")[0]
    if base_name in allowed_imports:
        module = __import__(name, globals, locals, fromlist, level)
        return module
    raise ImportError(f"Import not allowed: {name}")

safe_builtins["__import__"] = guarded_import
safe_globals = dict(__builtins__=safe_builtins)

code = compile_restricted(
    'from dateutil.parser import parse\ndt = parse("2024-01-01").strftime("%d/%m/%Y")'
)
safe_locals = dict()
exec(code, safe_globals, safe_locals)
print(safe_locals["dt"])