KeyError: ('dict', 'builtins') in `DiracRPCClient.py` preventing `webapp-daemon.py` startup

[asnaylor]

When running either webapp-daemon.py or monitoring-daemon.py I get KeyError: ('dict', 'builtins') in DiracRPCClient.py which prevents either daemon from running. I tested this on python 2.7 + 3.6 + 3.7

Here is the terminal Error:

$ user | ProductionSystem ⑂master: webapp-daemon.py start --debug-mode
[2019-06-20 13:53:32,296]   webapp-daemon :  WARNING : Config file '~/.config/productionsystem/productionsystem.conf' does not exist
Traceback (most recent call last):
  File "/Users/anaylor/anaconda2/envs/py36/bin/webapp-daemon.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/Users/anaylor/lzsim/lz_USDC/temp/docker_test/ProductionSystem/scripts/webapp-daemon.py", line 241, in <module>
    args.func(args)
  File "/Users/anaylor/lzsim/lz_USDC/temp/docker_test/ProductionSystem/scripts/webapp-daemon.py", line 77, in start
    WebApp = config_instance.entry_point_map['webapp']['daemon'].load()
  File "/Users/anaylor/anaconda2/envs/py36/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2434, in load
    return self.resolve()
  File "/Users/anaylor/anaconda2/envs/py36/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2440, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/anaylor/lzsim/lz_USDC/temp/docker_test/ProductionSystem/productionsystem/webapp/WebApp.py", line 13, in <module>
    from .services import (HTMLPageServer, CVMFSDirectoryListing, GitDirectoryListing,
  File "/Users/anaylor/lzsim/lz_USDC/temp/docker_test/ProductionSystem/productionsystem/webapp/services/__init__.py", line 5, in <module>
    from .CVMFSListing import CVMFSDirectoryListing
  File "/Users/anaylor/lzsim/lz_USDC/temp/docker_test/ProductionSystem/productionsystem/webapp/services/CVMFSListing.py", line 11, in <module>
    from productionsystem.apache_utils import check_credentials
  File "/Users/anaylor/lzsim/lz_USDC/temp/docker_test/ProductionSystem/productionsystem/apache_utils.py", line 17, in <module>
    from productionsystem.sql.models import Users
  File "/Users/anaylor/lzsim/lz_USDC/temp/docker_test/ProductionSystem/productionsystem/sql/models/__init__.py", line 11, in <module>
    ParametricJobs = ConfigSystem.get_instance().entry_point_map['dbmodels']['parametricjobs'].load()
  File "/Users/anaylor/anaconda2/envs/py36/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2434, in load
    return self.resolve()
  File "/Users/anaylor/anaconda2/envs/py36/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2440, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/anaylor/lzsim/lz_USDC/temp/docker_test/ProductionSystem/productionsystem/sql/models/ParametricJobs.py", line 26, in <module>
    from productionsystem.monitoring.diracrpc.DiracRPCClient import (dirac_api_client,
  File "/Users/anaylor/lzsim/lz_USDC/temp/docker_test/ProductionSystem/productionsystem/monitoring/diracrpc/DiracRPCClient.py", line 21, in <module>
    {}.__class__.__module__)]
KeyError: ('dict', 'builtins')

[alexanderrichards]

I can't reproduce this, this works fine for me... Can you check that the following works in a python session:

Python 3

>>> import rpyc
>>> rpyc.core.netref.builtin_classes_cache[('dict', 'builtins')]
<netref class 'builtins.dict'>

Python 2

import rpyc
rpyc.core.netref.builtin_classes_cache[('dict', '__builtin__')]
<netref class '__builtin__.dict'>

[asnaylor]

I just did a git pull on my ProductionSystem repo to update it, there are no modifications to any of the code. I use that repo just to check that ProductionSystems works with my python.

Python 3:

Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 12:04:33)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpyc
>>> rpyc.core.netref.builtin_classes_cache[('dict', 'builtins')]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: ('dict', 'builtins')
>>>

Python 2:

python
Python 2.7.15 |Anaconda custom (64-bit)| (default, Dec 14 2018, 13:10:39)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpyc
>>> rpyc.core.netref.builtin_classes_cache[('dict', '__builtin__')]
<netref class '__builtin__.dict'>

---

Andrew Naylor Astroparticle Physics PhD Student Department of Physics and Astronomy The University of Sheffield

Office:E18a, Hicks Building, Hounsfield Road, Sheffield S3 7RH, UK Email:andrew.naylor@sheffield.ac.uk mailto:andrew.naylor@sheffield.ac.uk

On 20 Jun 2019, at 14:16, Alexander Richards notifications@github.com wrote:

Also the monitoring / diracRPC modules should not be imported when running the webapp-daemon.py so this error should never be triggered. Have you made a modification such that the monitoring is imported?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/alexanderrichards/ProductionSystem/issues/49?email_source=notifications&email_token=AHYECYUUI5GFUGUL7XKBC6TP3N7JZA5CNFSM4HZSZWT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYFMEXQ#issuecomment-504021598, or mute the thread https://github.com/notifications/unsubscribe-auth/AHYECYRB3IMR4JZP2EQ6YRLP3N7JZANCNFSM4HZSZWTQ.

[alexanderrichards]

looks like a problem with the rpyc dependency. Have you installed within a virtualenv? If so can you scrap the env and start again with:

python -m virtualenv venv
. venv/bin/activate
pip install --upgrade pip setuptools wheel
pip install productionsystem@git+https://github.com/alexanderrichards/ProductionSystem.git

[asnaylor]

What’s unusual is my default python conda env (python 2.7) works fine with running webapp-daemon.py but when I do a clean build with Dockerfile which has worked previously [using python2.7-alpine3.8 as the os] I am getting this error now, so I am thinking something I have installed previously to my default env, I haven’t installed in my Dockerfile so this could be why I am getting that error

---

Andrew Naylor Astroparticle Physics PhD Student Department of Physics and Astronomy The University of Sheffield

Office:E18a, Hicks Building, Hounsfield Road, Sheffield S3 7RH, UK Email:andrew.naylor@sheffield.ac.uk mailto:andrew.naylor@sheffield.ac.uk

On 20 Jun 2019, at 14:24, Alexander Richards notifications@github.com wrote:

looks like a problem with the rpyc dependency. Have you installed within a virtualenv? If so can you scrap the env and start again with:

python -m virtualenv venv . venv/bin/activate pip install --upgrade pip setuptools wheel pip install productionsystem@git+https://github.com/alexanderrichards/ProductionSystem.git — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/alexanderrichards/ProductionSystem/issues/49?email_source=notifications&email_token=AHYECYVTQAWOGDSDO3AE5NDP3OAJ5A5CNFSM4HZSZWT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYFM25I#issuecomment-504024437, or mute the thread https://github.com/notifications/unsubscribe-auth/AHYECYSDP74NBQT7PXU27EDP3OAJ5ANCNFSM4HZSZWTQ.

[alexanderrichards]

out of curiosity what do you have as keys in Python 3:

list(rpyc.core.netref.builtin_classes_cache.keys())

[georgemm01]

Hello, That's exactly what I did to sort this out. The format is different (for whatever reason): This is what I did to solve this issue: rpyc.core.netref.builtin_classes_cache[{}.class.module+'.'+{}.class.name]

And this is what the keys show:

import rpyc rpyc.core.netref.builtin_classes_cache.keys() dict_keys(['builtins.Exception', 'builtins.generator', 'builtins.object', 'builtins.frame', 'builtins.tuple', 'builtins.list_iterator', 'builtins.set', 'builtins.module', 'builtins.code', 'builtins.BaseException', 'builtins.frozenset', 'builtins.NoneType', 'builtins.builtin_function_or_method', 'builtins.type', 'builtins.wrapper_descriptor', 'builtins.range_iterator', 'builtins.method', 'builtins.float', 'builtins.int', 'builtins.slice', 'builtins.str', 'builtins.method-wrapper', 'builtins.tuple_iterator', 'builtins.list', 'builtins.memoryview', 'builtins.traceback', 'builtins.bytes', 'builtins.bool', 'builtins.complex', 'builtins.bytearray', 'builtins.function', 'builtins.set_iterator', 'builtins.dict'])

[alexanderrichards]

must be a change in rpyc. What version of rpyc do you have?

[asnaylor]

I figure out the issue is the version of rpyc is different for these different versions. The code works with version rpyc==3.4.4 but it was given me the error for rpyc==4.1.0

[alexanderrichards]

ah ha, I'm on 4.0.2 in Python 3.6.2. Clearly this is a new change (between 4.0.2 and 4.1.0) hence Python 2 version of rpyc still works. Thanks for bringing this to my attention. I'll get a patch fix ready.

[alexanderrichards]

Ok fix with full backwards compatibility incoming in PR 50. If you have a chance I would appreciate if you can try that branch and see if it solves your problems. It seems to work for me.

[asnaylor]

I tried it with my docker setup and it failed with both. It's using rpyc-4.1.0 when installing with pip. Python 2.7

 File "/website/ProductionSystem/productionsystem/monitoring/diracrpc/DiracRPCClient.py", line 21, in <module>
    netref_dict = netref_cache.get('builtins.dict', netref_cache[('dict', {}.__class__.__module__)])
KeyError: (u'dict', '__builtin__')

Python 3.6

  File "/website/ProductionSystem/productionsystem/monitoring/diracrpc/DiracRPCClient.py", line 21, in <module>
    netref_dict = netref_cache.get('builtins.dict', netref_cache[('dict', {}.__class__.__module__)])
KeyError: ('dict', 'builtins')

[alexanderrichards]

ok the Python 2 error is due to unicode key. Thats an easy fix.

For Python 3.6 it looks like it hasn't found the builtins.dict key. Did I make a typo? what does the following give you?

'builtins.dict' in rpyc.core.netref.builtin_classes_cache

[asnaylor]

ah okay, for python3.6 i get:

import rpyc 'builtins.dict' in rpyc.core.netref.builtin_classes_cache False

[alexanderrichards]

What keys do you have in that dict?

[asnaylor]

Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 12:04:33)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpyc
>>> rpyc.core.netref.builtin_classes_cache.keys()
dict_keys([('type', 'builtins'), ('object', 'builtins'), ('bool', 'builtins'), ('complex', 'builtins'), ('dict', 'builtins'), ('float', 'builtins'), ('int', 'builtins'), ('list', 'builtins'), ('slice', 'builtins'), ('str', 'builtins'), ('tuple', 'builtins'), ('set', 'builtins'), ('frozenset', 'builtins'), ('Exception', 'builtins'), ('NoneType', 'builtins'), ('builtin_function_or_method', 'builtins'), ('generator', 'builtins'), ('method', 'builtins'), ('code', 'builtins'), ('frame', 'builtins'), ('traceback', 'builtins'), ('module', 'builtins'), ('function', 'builtins'), ('wrapper_descriptor', 'builtins'), ('method-wrapper', 'builtins'), ('list_iterator', 'builtins'), ('tuple_iterator', 'builtins'), ('set_iterator', 'builtins'), ('BaseException', 'builtins'), ('bytes', 'builtins'), ('bytearray', 'builtins'), ('range_iterator', 'builtins'), ('memoryview', 'builtins')])
>>>

[alexanderrichards]

So ('dict', 'builtins') is a key, I wonder why then your getting KeyError?

What about:

('dict', 'builtins') in rpyc.core.netref.builtin_classes_cache

[asnaylor]

yep it appears:

>>> import rpyc
>>> ('dict', 'builtins') in rpyc.core.netref.builtin_classes_cache
True

[alexanderrichards]

Then barring a typo which it doesn't look like I've made, the original KeyError can't be right. What about if you repeat the above, copying the key from the KeyError message to see if there is a typo

[asnaylor]

Ah sorry alex, when i was testing the ('dict', 'builtins') in rpyc.core.netref.builtin_classes_cache I realise I was doing it with a build of python 3.6 which had rpyc==3.4.4 and when i reinstalled it again with pip install -e . it works now but when i use the docker setup to run ProductionSystems i still get the KeyError.

In docker i ran the key check [rpyc==4.1.0]:

Python 3.6.8 (default, Mar  7 2019, 22:52:35)
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpyc
>>> rpyc.core.netref.builtin_classes_cache.keys()
dict_keys(['builtins.type', 'builtins.object', 'builtins.bool', 'builtins.complex', 'builtins.dict', 'builtins.float', 'builtins.int', 'builtins.list', 'builtins.slice', 'builtins.str', 'builtins.tuple', 'builtins.set', 'builtins.frozenset', 'builtins.Exception', 'builtins.NoneType', 'builtins.builtin_function_or_method', 'builtins.generator', 'builtins.method', 'builtins.code', 'builtins.frame', 'builtins.traceback', 'builtins.module', 'builtins.function', 'builtins.wrapper_descriptor', 'builtins.method-wrapper', 'builtins.list_iterator', 'builtins.tuple_iterator', 'builtins.set_iterator', 'builtins.BaseException', 'builtins.bytes', 'builtins.bytearray', 'builtins.range_iterator', 'builtins.memoryview'])
>>> ('dict', 'builtins') in rpyc.core.netref.builtin_classes_cache
False
>>> rpyc.core.netref.builtin_classes_cache[('dict', 'builtins')]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: ('dict', 'builtins')
>>> rpyc.core.netref.builtin_classes_cach
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'rpyc.core.netref' has no attribute 'builtin_classes_cach'

So for rpyc-4.1.0 it doesn't have it.

[alexanderrichards]

So for rpyc-4.1.0 it doesn't have it.

Indeed, but this should now work with the patch that I have on the branch ready to commit. I'm guessing that this docker test was using the master as it looks like it's still running the old code. When you re-setup did you explicitly name the branch for the patch otherwise you will get master by default.

[asnaylor]

Yeah i did git clone -b rpyc_netref_cache_fix git@github.com:alexanderrichards/ProductionSystem.git and in the dockerfile i copy across the whole ProductionSystems repo and do a pip install -e . inside that folder.

[alexanderrichards]

You can simply do the following in the docker file

pip install productionsystem@git+https://github.com/alexanderrichards/ProductionSystem.git@rpyc_netref_cache_fix 

[alexanderrichards]

The problem you report is because your version of rpyc has the new keys which should have been picked up by the new code. What is the traceback you get from running the code

[asnaylor]

I tried it with pip install productionsystem@git+https://github.com/alexanderrichards/ProductionSystem.git@rpyc_netref_cache_fix but i still get the same error:

[2019-06-21 11:32:26,686] monitoring-daemon :  WARNING : Config file '~/.config/productionsystem/productionsystem.conf' does not exist
[2019-06-21 11:32:26,696]   webapp-daemon :  WARNING : Config file '~/.config/productionsystem/productionsystem.conf' does not exist
Traceback (most recent call last):
  File "/usr/local/bin/webapp-daemon.py", line 241, in <module>
    args.func(args)
  File "/usr/local/bin/webapp-daemon.py", line 71, in start
    apache_utils = importlib.import_module('productionsystem.apache_utils')
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.6/site-packages/productionsystem/apache_utils.py", line 17, in <module>
    from productionsystem.sql.models import Users
  File "/usr/local/lib/python3.6/site-packages/productionsystem/sql/models/__init__.py", line 11, in <module>
    ParametricJobs = ConfigSystem.get_instance().entry_point_map['dbmodels']['parametricjobs'].load()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2434, in load
    return self.resolve()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2440, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.6/site-packages/productionsystem/sql/models/ParametricJobs.py", line 26, in <module>
    from productionsystem.monitoring.diracrpc.DiracRPCClient import (dirac_api_client,
  File "/usr/local/lib/python3.6/site-packages/productionsystem/monitoring/diracrpc/DiracRPCClient.py", line 21, in <module>
    netref_dict = netref_cache.get('builtins.dict', netref_cache[('dict', {}.__class__.__module__)])
KeyError: ('dict', 'builtins')

[alexanderrichards]

File "/usr/local/lib/python3.6/site-packages/productionsystem/monitoring/diracrpc/DiracRPCClient.py", line 21, in netref_dict = netref_cache.get('builtins.dict', netref_cache[('dict', {}.class.module)]) KeyError: ('dict', 'builtins')

This confuses me as presumably with the new rpyc you will get True for:

"builtins.dict" in rpyc.core.netref.builtin_classes_cache

Can you just check this. If so then I bet whats happening is that it's evaluating the __getitem__ before attempting the get

[asnaylor]

Yeah it came up True

Python 3.6.8 (default, Mar  7 2019, 22:52:35)
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from builtins import *
>>> import rpyc
>>> "builtins.dict" in rpyc.core.netref.builtin_classes_cache
True

[alexanderrichards]

ok I think I understand this now. I'll update the patch

[alexanderrichards]

ok patch updated, please try again now.

[asnaylor]

great it works with python 3.6 but i'm still getting an error with python 2.7

Traceback (most recent call last):
  File "/usr/local/bin/webapp-daemon.py", line 241, in <module>
    args.func(args)
  File "/usr/local/bin/webapp-daemon.py", line 71, in start
    apache_utils = importlib.import_module('productionsystem.apache_utils')
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/local/lib/python2.7/site-packages/productionsystem/apache_utils.py", line 17, in <module>
    from productionsystem.sql.models import Users
  File "/usr/local/lib/python2.7/site-packages/productionsystem/sql/models/__init__.py", line 11, in <module>
    ParametricJobs = ConfigSystem.get_instance().entry_point_map['dbmodels']['parametricjobs'].load()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2434, in load
    return self.resolve()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2440, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python2.7/site-packages/productionsystem/sql/models/ParametricJobs.py", line 26, in <module>
    from productionsystem.monitoring.diracrpc.DiracRPCClient import (dirac_api_client,
  File "/usr/local/lib/python2.7/site-packages/productionsystem/monitoring/diracrpc/DiracRPCClient.py", line 26, in <module>
    netref_dict = netref_cache[(native_str('dict'), {}.__class__.__module__)]
KeyError: ('dict', '__builtin__')

[alexanderrichards]

Strange, can you once again do the following in python 2.7

("dict", "__builtin__") in rpyc.core.netref.builtin_classes_cache

print rpyc.core.netref.builtin_classes_cache.keys()

[asnaylor]

Python 2.7.15 (default, Dec  8 2018, 02:35:22)
[GCC 6.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpyc
>>> ("dict", "__builtin__") in rpyc.core.netref.builtin_classes_cache
False
>>> print rpyc.core.netref.builtin_classes_cache.keys()
['__builtin__.basestring', '__builtin__.slice', '__builtin__.generator', '__builtin__.code', '__builtin__.rangeiterator', '__builtin__.frozenset', '__builtin__.dict', '__builtin__.instance', 'exceptions.Exception', '__builtin__.object', '__builtin__.int', '__builtin__.file', '__builtin__.str', '__builtin__.complex', '__builtin__.frame', '__builtin__.float', '__builtin__.xrange', '__builtin__.listiterator', '__builtin__.set', '__builtin__.function', '__builtin__.tupleiterator', '__builtin__.NoneType', '__builtin__.long', '__builtin__.tuple', '__builtin__.instancemethod', '__builtin__.dictproxy', '__builtin__.list', '__builtin__.setiterator', 'exceptions.BaseException', '__builtin__.module', '__builtin__.wrapper_descriptor', '__builtin__.builtin_function_or_method', '__builtin__.unicode', '__builtin__.traceback', '__builtin__.type', '__builtin__.classobj', '__builtin__.bool', '__builtin__.method-wrapper']

[alexanderrichards]

Oh goodness they have made the new rpyc version also for python 2. Right o, one more iteration

[alexanderrichards]

While I fix the patch can you tell me the version number of rpyc under python 2.7. is it also 4.1.0?

[asnaylor]

yep rpyc==4.1.0 for both py3.6 and 2.7

[alexanderrichards]

Thanks, I'll sort the patch out probably Monday now since it's 9pm here :wink:

[alexanderrichards]

Right, rather than try and maintain the matrix of Python 2.7/Python3 vs rpyc < 4.1.0/rpyc>4.1.0 I have decided that it's far cleaner in the code to just work with the new rpyc format and require rpyc >=4.1.0.

Since they have also released this version for Python 2.7 this should all be fine and the code can go back to almost how it was originally just with the new key syntax.

I have now checked this into the branch. Please give it a final check that it's working for you in 2.7 and 3. Thanks again for bringing this to my attention.

[asnaylor]

Great, thanks alex. It works for both 2.7 and 3.6 without any issues.