tracebackrepy.py triggers "Unsafe call" error in safe.py

[aaaaalbert]

As a consequence of #69, safe.py destroying builtins like import for everything that runs after it (including other Python code of the Repy runtime!), tracebackrepy.py triggers an error when trying to log an "Internal Error" of Repy into the servicelogger log. See below for a traceback. This is because it imports a library inside of the handle_internalerror function rather than in the outmost scope (where it would be executed when tracebackrepy itself is imported).

Since it's probably a good idea to keep safe.py as paranoid about builtins as it is, the workaround will be to move the import of servicelogger out of the function.

@DennisMirante suggested a patch whose gist is to move the import to the top,

import servicelogger
service_logger = servicelogger
del servicelogger

and then use service_logger in place of servicelogger in handle_internalerrror.

(With this, we need to check that service_logger is not callable from within Repy code.)

---

Here's an emanation of the problem, as reported by @yyzhuang.

---
Uncaught exception!

---
Following is a full traceback, and a user traceback.
The user traceback excludes non-user modules. The most recent call is
displayed last.

Full debugging traceback:
 "repyV2/repy.py", line 177, in execute_namespace_until_completion
"/storage/emulated/0/Android/data/com.sensibilitytestbed/files/sl4a/seattle/seattle_repy/repyV2/virtual_namespace.py",
line 117, in evaluate
"/storage/emulated/0/Android/data/com.sensibilitytestbed/files/sl4a/seattle/seattle_repy/repyV2/safe.py",
line 588, in safe_run
 "superfriend.r2py", line 3551, in <module>
 "superfriend.r2py", line 3487, in main

"/storage/emulated/0/Android/data/com.sensibilitytestbed/files/sl4a/seattle/seattle_repy/repyV2/namespace.py",
line 1220, in wrapped_function
"/storage/emulated/0/Android/data/com.sensibilitytestbed/files/sl4a/seattle/seattle_repy/repyV2/namespace.py",
line 218, in _handle_internalerror
"/storage/emulated/0/Android/data/com.sensibilitytestbed/files/sl4a/seattle/seattle_repy/repyV2/tracebackrepy.py",
line 209, in handle_internalerror
"/storage/emulated/0/Android/data/com.sensibilitytestbed/files/sl4a/seattle/seattle_repy/repyV2/safe.py",
line 493, in exceptionraiser

User traceback:
 "superfriend.r2py", line 3551, in <module>
 "superfriend.r2py", line 3487, in main

Unsafe call: ("Unsafe call '__import__' with args '('servicelogger',
{'handle_internalerror': <function handle_internalerror at 0x40751130>,
'sys': <module 'sys' (built-in)>, 'servicelog': True, 'TB_SKIP_MODULES':
['repy.py', 'safe.py', 'virtual_namespace.py', 'namespace.py',
'emulcomm.py', 'emultimer.py', 'emulmisc.py', 'emulfile.py',
'nonportable.py', 'socket.py'], 'logdirectory':
'/storage/emulated/0/Android/data/com.sensibilitytestbed/files/sl4a/seattle/seattle_repy/repyV2',
'__builtins__': {'bytearray': <function exceptionraiser at 0x407a6730>,
'IndexError': <type 'exceptions.IndexError'>, 'all': <function
exceptionraiser at 0x407a6eb0>, 'help': <function exceptionraiser at
0x407a6fb0>, 'vars': <function exceptionraiser at 0x407a6630>,
'SyntaxError': <type 'exceptions.SyntaxError'>, 'CodeUnsafeError': <class
'exception_hierarchy.CodeUnsafeError'>, 'unicode': <function
exceptionraiser at 0x407a6670>, 'UnicodeDecodeError': <type
'exceptions.UnicodeDecodeError'>, 'memoryview': <function exceptionraiser
at 0x407a66b0>, 'isinstance': <built-in function isinstance>,
'ContextUnsafeError': <class 'exception_hierarchy.ContextUnsafeError'>,
'AlreadyListeningError': <class
'exception_hierarchy.AlreadyListeningError'>, 'copyright': '',
'CleanupInProgressError': <class
'exception_hierarchy.CleanupInProgressError'>, 'NameError': <type
'exceptions.NameError'>, 'BytesWarning': <function exceptionraiser at
0x407a66f0>, 'dict': <type 'dict'>, 'input': <function exceptionraiser at
0x407a6770>, 'SocketWouldBlockError': <class
'exception_hierarchy.SocketWouldBlockError'>, 'oct': <built-in function
oct>, 'bin': <function exceptionraiser at 0x407a67b0>, 'SystemExit': <type
'exceptions.SystemExit'>, 'SeekPastEndOfFileError': <class
'exception_hierarchy.SeekPastEndOfFileError'>, 'StandardError': <type
'exceptions.StandardError'>, 'format': <function exceptionraiser at
0x407a67f0>, 'repr': <built-in function repr>, 'SocketClosedRemote': <class
'exception_hierarchy.SocketClosedRemote'>, 'sorted': <function
exceptionraiser at 0x407a6830>, 'SafeDict': <function get_SafeDict at
0x40776d70>, 'False': False, 'RuntimeWarning': <type
'exceptions.RuntimeWarning'>, 'list': <type 'list'>, 'iter': <function
exceptionraiser at 0x407a6870>, 'reload': <function exceptionraiser at
0x407a68b0>, 'Warning': <type 'exceptions.Warning'>, 'AddressBindingError':
<class 'exception_hierarchy.AddressBindingError'>, '__package__': <function
exceptionraiser at 0x407a6e70>, 'FileNotFoundError': <class
'exception_hierarchy.FileNotFoundError'>, 'RepyException': <class
'exception_hierarchy.RepyException'>, 'FileInUseError': <class
'exception_hierarchy.FileInUseError'>, 'round': <built-in function round>,
'dir': <function exceptionraiser at 0x407a6ef0>, 'cmp': <built-in function
cmp>, 'TCPServerSocketInvalidError': <class
'exception_hierarchy.TCPServerSocketInvalidError'>, 'set': <type 'set'>,
'bytes': <function exceptionraiser at 0x407a6f30>,
'ResourceExhaustedError': <class
'exception_hierarchy.ResourceExhaustedError'>, 'reduce': <built-in function
reduce>, 'intern': <function exceptionraiser at 0x407a6f70>, 'issubclass':
<built-in function issubclass>, 'Ellipsis': Ellipsis, 'EOFError': <type
'exceptions.EOFError'>, 'ResourceForbiddenError': <class
'exception_hierarchy.ResourceForbiddenError'>, 'locals': <function
exceptionraiser at 0x407aa3f0>, 'BufferError': <function exceptionraiser at
0x407aab30>, 'slice': <type 'slice'>, 'SocketClosedLocal': <class
'exception_hierarchy.SocketClosedLocal'>, 'FloatingPointError': <type
'exceptions.FloatingPointError'>, 'sum': <built-in function sum>,
'getattr': <function exceptionraiser at 0x407aa4f0>, 'abs': <built-in
function abs>, 'exit': <function exceptionraiser at 0x407aa530>, 'print':
<function exceptionraiser at 0x407aa570>, 'True': True, 'FutureWarning':
<type 'exceptions.FutureWarning'>, 'ImportWarning': <function
exceptionraiser at 0x407aa5b0>, 'None': None, 'hash': <function
exceptionraiser at 0x407aa5f0>, 'ReferenceError': <type
'exceptions.ReferenceError'>, 'len': <built-in function len>, 'credits':
'', 'frozenset': <type 'frozenset'>, '__name__': '', 'ord': <built-in
function ord>, 'super': <function exceptionraiser at 0x407aa2b0>,
'TypeError': <type 'exceptions.TypeError'>, 'license': '',
'KeyboardInterrupt': <type 'exceptions.KeyboardInterrupt'>, 'UserWarning':
<type 'exceptions.UserWarning'>, 'filter': <built-in function filter>,
'range': <built-in function range>, 'staticmethod': <function
exceptionraiser at 0x407aa330>, 'SystemError': <type
'exceptions.SystemError'>, 'BaseException': <type
'exceptions.BaseException'>, 'pow': <built-in function pow>,
'RuntimeError': <type 'exceptions.RuntimeError'>, 'float': <type 'float'>,
'MemoryError': <type 'exceptions.MemoryError'>, 'StopIteration': <type
'exceptions.StopIteration'>, 'globals': <function exceptionraiser at
0x407aa4b0>, 'divmod': <built-in function divmod>, 'enumerate': <function
exceptionraiser at 0x407aa7f0>, 'apply': <function exceptionraiser at
0x407aabf0>, 'LookupError': <type 'exceptions.LookupError'>, 'open':
<function exceptionraiser at 0x407aa6f0>, 'quit': <function exceptionraiser
at 0x407aa6b0>, 'basestring': <function exceptionraiser at 0x407aa9f0>,
'UnicodeError': <type 'exceptions.UnicodeError'>, 'zip': <built-in function
zip>, 'hex': <built-in function hex>, 'long': <type 'long'>, 'next':
<function exceptionraiser at 0x407aa030>, 'ImportError': <type
'exceptions.ImportError'>, 'chr': <built-in function chr>,
'ConnectionRefusedError': <class
'exception_hierarchy.ConnectionRefusedError'>, 'xrange': <type 'xrange'>,
'FileError': <class 'exception_hierarchy.FileError'>, 'type': <function
safe_type at 0x40751d30>, '__doc__': '', 'Exception': <type
'exceptions.Exception'>, 'TimeoutError': <class
'exception_hierarchy.TimeoutError'>, 'tuple': <type 'tuple'>,
'UnicodeTranslateError': <type 'exceptions.UnicodeTranslateError'>,
'reversed': <function exceptionraiser at 0x407aa070>,
'InternetConnectivityError': <class
'exception_hierarchy.InternetConnectivityError'>, 'UnicodeEncodeError':
<type 'exceptions.UnicodeEncodeError'>, 'IOError': <type
'exceptions.IOError'>, 'hasattr': <function exceptionraiser at 0x407aa0b0>,
'delattr': <function exceptionraiser at 0x407aa0f0>, 'setattr': <function
exceptionraiser at 0x407aa130>, 'raw_input': <function exceptionraiser at
0x407aa170>, 'SyntaxWarning': <type 'exceptions.SyntaxWarning'>, 'compile':
<function exceptionraiser at 0x407aa1b0>, 'ArithmeticError': <type
'exceptions.ArithmeticError'>, 'ResourceUsageError': <class
'exception_hierarchy.ResourceUsageError'>, 'str': <type 'str'>, 'property':
<function exceptionraiser at 0x407aa1f0>, 'GeneratorExit': <function
exceptionraiser at 0x407aa230>, 'FileClosedError': <class
'exception_hierarchy.FileClosedError'>, 'int': <type 'int'>, '__import__':
<function exceptionraiser at 0x407aa270>, 'KeyError': <type
'exceptions.KeyError'>, 'coerce': <function exceptionraiser at 0x407aa2f0>,
'PendingDeprecationWarning': <type 'exceptions.PendingDeprecationWarning'>,
'file': <function exceptionraiser at 0x407aa370>, 'EnvironmentError': <type
'exceptions.EnvironmentError'>, 'unichr': <function exceptionraiser at
0x407aa3b0>, 'id': <built-in function id>, 'OSError': <type
'exceptions.OSError'>, 'DeprecationWarning': <type
'exceptions.DeprecationWarning'>, 'RepyArgumentError': <class
'exception_hierarchy.RepyArgumentError'>, 'min': <built-in function min>,
'UnicodeWarning': <function exceptionraiser at 0x407aa430>,
'LocalIPChanged': <class 'exception_hierarchy.LocalIPChanged'>, 'execfile':
<function exceptionraiser at 0x407aa470>, 'any': <function exceptionraiser
at 0x407aa630>, 'complex': <type 'complex'>, 'bool': <type 'bool'>,
'LockDoubleReleaseError': <class
'exception_hierarchy.LockDoubleReleaseError'>, 'DuplicateTupleError':
<class 'exception_hierarchy.DuplicateTupleError'>, 'ValueError': <type
'exceptions.ValueError'>, 'NotImplemented': NotImplemented, 'map':
<built-in function map>, 'buffer': <function exceptionraiser at
0x407aa670>, 'max': <built-in function max>, 'object': <type 'object'>,
'NetworkError': <class 'exception_hierarchy.NetworkError'>, 'TabError':
<type 'exceptions.TabError'>, 'callable': <function exceptionraiser at
0x407aa730>, 'ZeroDivisionError': <type 'exceptions.ZeroDivisionError'>,
'eval': <function exceptionraiser at 0x407aa770>, 'NetworkAddressError':
<class 'exception_hierarchy.NetworkAddressError'>, '__debug__': True,
'IndentationError': <type 'exceptions.IndentationError'>, 'AssertionError':
<type 'exceptions.AssertionError'>, 'classmethod': <function
exceptionraiser at 0x407aa7b0>, 'UnboundLocalError': <type
'exceptions.UnboundLocalError'>, 'NotImplementedError': <type
'exceptions.NotImplementedError'>, 'AttributeError': <type
'exceptions.AttributeError'>, 'OverflowError': <type
'exceptions.OverflowError'>}, 'os': <module 'os' from
'/storage/emulated/0/Android/data/com.sensibilitytestbed/files/extras/python/os.pyc'>,
'__file__':
'/storage/emulated/0/Android/data/com.sensibilitytestbed/files/sl4a/seattle/seattle_repy/repyV2/tracebackrepy.pyc',
'traceback': <module 'traceback' from
'/storage/emulated/0/Android/data/com.sensibilitytestbed/files/extras/python/traceback.pyc'>,
'handle_exception': <function handle_exception at 0x407510f0>,
'__package__': None, 'fakelinecache': <module 'fakelinecache' from
'/storage/emulated/0/Android/data/com.sensibilitytestbed/files/sl4a/seattle/seattle_repy/repyV2/fakelinecache.pyc'>,
'format_exception': <function format_exception at 0x407510b0>, 'harshexit':
<module 'harshexit' from
'/storage/emulated/0/Android/data/com.sensibilitytestbed/files/sl4a/seattle/seattle_repy/repyV2/harshexit.pyc'>,
'initialize': <function initialize at 0x40751070>, '__name__':
'tracebackrepy', 'exception_hierarchy': <module 'exception_hierarchy' from
'/storage/emulated/0/Android/data/com.sensibilitytestbed/files/sl4a/seattle/seattle_repy/repyV2/exception_hierarchy.pyc'>,
'__doc__': ' \\nAuthor: Justin Cappos\\n\\nStart Date: September 17th,
2008\\n\\nDescription:\\nModule for printing clean tracebacks.   It takes
the python traceback and \\nmakes the output look nicer so the programmer
can tell what is happening...\\n\\n'}, None, None)', kwargs '{}'",)

---

[aaaaalbert]

To test @DennisMirante's patch, I modified the Repy runtime function randombytes (any other will do too) to call tracebackrepy.handle_internalerror. This now reliably triggers an "Internal Error" when I run the trigger script from the command line.

To trigger the "Unsafe call" problem, we need to make Repy use the servicelog: python repy.py --servicelog restrictions.default trigger_internal_error.r2py

This however only shifts the problem to the next unsafe call, this time it's hasattr in the Python posixpath library:

Internal Error
---
Uncaught exception!
None
Inner abort of servicelogger
("Unsafe call 'hasattr' with args '(<module 'posixpath' from '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc'>, 'samefile')', kwargs '{}'",) <class 'exception_hierarchy.RunBuiltinException'>
Traceback (most recent call last):
  File "/Users/albert/Downloads/sensibility-20140924-1449/seattle_repy/tracebackrepy.py", line 246, in handle_internalerror
  File "/Users/albert/Downloads/sensibility-20140924-1449/seattle_repy/servicelogger.py", line 213, in multi_process_log
  File "/Users/albert/Downloads/sensibility-20140924-1449/seattle_repy/servicelogger.py", line 119, in get_servicevessel
  File "/Users/albert/Downloads/sensibility-20140924-1449/seattle_repy/persist.py", line 163, in restore_object
  File "/Users/albert/Downloads/sensibility-20140924-1449/seattle_repy/persist.py", line 89, in _copy
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 116, in copy
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 67, in copyfile
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 55, in _samefile
  File "/Users/albert/Downloads/sensibility-20140924-1449/seattle_repy/safe.py", line 493, in exceptionraiser
RunBuiltinException: ("Unsafe call 'hasattr' with args '(<module 'posixpath' from '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc'>, 'samefile')', kwargs '{}'",)

[JustinCappos]

If you add: "hasattr = hasattr" to the outer scope of the servicelogger, I think it should fix this issue. We have similar code elsewhere that stashes references to exactly the unsafe calls we need.

On Fri, Sep 26, 2014 at 5:52 PM, aaaaalbert notifications@github.com wrote:

To test @DennisMirante https://github.com/DennisMirante's patch, I modified the Repy runtime function randombytes (any other will do too) to call tracebackrepy.handle_internalerror. This now reliably triggers an "Internal Error" when I run the trigger script from the command line.

To trigger the "Unsafe call" problem, we need to make Repy use the servicelog: python repy.py --servicelog restrictions.default trigger_internal_error.r2py

This however only shifts the problem to the next unsafe call, this time it's hasattr in the Python posixpath library:

Internal Error

Uncaught exception! None Inner abort of servicelogger ("Unsafe call 'hasattr' with args '(<module 'posixpath' from '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc'>, 'samefile')', kwargs '{}'",) <class 'exception_hierarchy.RunBuiltinException'> Traceback (most recent call last): File "/Users/albert/Downloads/sensibility-20140924-1449/seattle_repy/tracebackrepy.py", line 246, in handle_internalerror File "/Users/albert/Downloads/sensibility-20140924-1449/seattle_repy/servicelogger.py", line 213, in multi_process_log File "/Users/albert/Downloads/sensibility-20140924-1449/seattle_repy/servicelogger.py", line 119, in get_servicevessel File "/Users/albert/Downloads/sensibility-20140924-1449/seattle_repy/persist.py", line 163, in restore_object File "/Users/albert/Downloads/sensibility-20140924-1449/seattle_repy/persist.py", line 89, in _copy File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 116, in copy File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 67, in copyfile File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 55, in _samefile File "/Users/albert/Downloads/sensibility-20140924-1449/seattle_repy/safe.py", line 493, in exceptionraiser RunBuiltinException: ("Unsafe call 'hasattr' with args '(<module 'posixpath' from '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc'>, 'samefile')', kwargs '{}'",)

— Reply to this email directly or view it on GitHub https://github.com/SeattleTestbed/repy_v2/issues/92#issuecomment-57025409 .

[aaaaalbert]

I ended up adding a couple of things to namespace.py:

import shutil
shutil.hasattr = hasattr
shutil.open = open

import persist
persist.open = open
persist.eval = eval

I'll experiment some more to see if that's the best place to put these additional stashes.