ModuleWatcher: ability to append to whitelist

philipstarkey commented 5 years ago

Original report (archived issue) by Russell Anderson (Bitbucket: rpanderson, GitHub: rpanderson).

Use case: reloading tensorflow fails.

Minimum failing example: lyse analysis routine containing:

import your_face
import tensorflow.core

… where your_face.py contains:

print('Your face')

Modify your_face.py to trigger module reloading, and get:

Traceback (most recent call last):
  File "C:\labscript_suite\userlib\analysislib\common\tensorflow_bug.py", line 2, in <module>
    import tensorflow.core
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\__init__.py", line 24, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\__init__.py", line 52, in <module>
    from tensorflow.core.framework.graph_pb2 import *
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\core\framework\graph_pb2.py", line 15, in <module>
    from tensorflow.core.framework import node_def_pb2 as tensorflow_dot_core_dot_framework_dot_node__def__pb2
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\core\framework\node_def_pb2.py", line 15, in <module>
    from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\core\framework\attr_value_pb2.py", line 15, in <module>
    from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\core\framework\tensor_pb2.py", line 15, in <module>
    from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\core\framework\resource_handle_pb2.py", line 91, in <module>
    __module__ = 'tensorflow.core.framework.resource_handle_pb2'
TypeError: A Message class can only inherit from Message

Similar issue reported here, where Spyder users report disabling the user-module reloader, or preventing google.* modules from being reloaded. For me, the above error was resolved by modifying modulewatcher.py to prevent tensorflow.* modules from being reloaded.

philipstarkey commented 5 years ago

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

What do you think about having ModuleWatcher by default whitelist all modules that are in site-packages/standard library? That way it will only ever reload 'user' modules, and any of these complex modules that do magical things and cause problems will never be reloaded. And if they're in site-packages then chances are the user is not modifying them, and if they are they know what they are doing. Possible issue: the user upgrades a package using pip or conda, it would then not be reloaded.

Alternately, ModuleWatcher could reload all modules if it detects a change on disk in a module in site-packages, and it could reload only 'user' modules if it detects a change in a module outside site-packages. This would be 90% of the way there without user action required.

philipstarkey commented 5 years ago

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

Have modulewatcher whitelist all modules within standard package directories. This way, only 'user' code will be unloaded or monitored for changes.

This addresses issue #23, though it is different to the solution than proposed there.

→ \<\<cset c3141fb6587a1673aca0716a471527c3bf65b56c>>

labscript-suite-oldfinal1 / labscript_utils

ModuleWatcher: ability to append to whitelist #23