Cannot serialize entries when implementing a custom session backend

[alehuo]

I'm currently implementing a custom Redis-based session backend for use with the pyoidc consumer. However, when trying to serialize some key-value pairs, I get an error which is shown below:

app  | [2020-01-17 11:40:27 +0000] [7] [INFO] Starting gunicorn 20.0.4
app  | [2020-01-17 11:40:27 +0000] [7] [INFO] Listening at: http://0.0.0.0:5000 (7)
app  | [2020-01-17 11:40:27 +0000] [7] [INFO] Using worker: sync
app  | [2020-01-17 11:40:27 +0000] [10] [INFO] Booting worker with pid: 10
nginx_1        | 172.30.0.1 - - [17/Jan/2020:11:42:10 +0000] "GET / HTTP/1.1" 200 849 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"
nginx_1        | 172.30.0.1 - - [17/Jan/2020:11:42:36 +0000] "GET / HTTP/1.1" 200 849 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"
nginx_1        | 172.30.0.1 - - [17/Jan/2020:11:42:40 +0000] "GET /auth HTTP/1.1" 302 822 "http://localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"
nginx_1        | 172.30.0.1 - - [17/Jan/2020:11:42:44 +0000] "GET /auth/callback?code=*******&state==******* HTTP/1.1" 500 141 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"
app  | [2020-01-17 11:42:44 +0000] [10] [ERROR] Error handling request /auth/callback?code==*******&state==*******
app  | Traceback (most recent call last):
app  |   File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 134, in handle
app  |     self.handle_request(listener, req, client, addr)
app  |   File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 175, in handle_request
app  |     respiter = self.wsgi(environ, resp.start_response)
app  |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2463, in __call__
app  |     return self.wsgi_app(environ, start_response)
app  |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2449, in wsgi_app
app  |     response = self.handle_exception(e)
app  |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1866, in handle_exception
app  |     reraise(exc_type, exc_value, tb)
app  |   File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
app  |     raise value
app  |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
app  |     response = self.full_dispatch_request()
app  |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
app  |     rv = self.handle_user_exception(e)
app  |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
app  |     reraise(exc_type, exc_value, tb)
app  |   File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
app  |     raise value
app  |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
app  |     rv = self.dispatch_request()
app  |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
app  |     return self.view_functions[rule.endpoint](**req.view_args)
app  |   File "/app/src/app.py", line 58, in auth_callback
app  |     consumer.complete(state=aresp["state"])
app  |   File "/usr/local/lib/python3.6/site-packages/oic/oic/consumer.py", line 475, in complete
app  |     self._backup(state)
app  |   File "/usr/local/lib/python3.6/site-packages/oic/oic/consumer.py", line 231, in _backup
app  |     self.sdb[sid] = self.dictionary()
app  |   File "/app/src/RedisSessionBackend.py", line 23, in __setitem__
app  |     self.r.set(key, jsonpickle.encode(value))
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 96, in encode
app  |     return backend.encode(context.flatten(value, reset=reset))
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 214, in flatten
app  |     return self._flatten(obj)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 218, in _flatten
app  |     return self._pop(self._flatten_obj(obj))
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 239, in _flatten_obj
app  |     return flatten_func(obj)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 503, in _flatten_dict_obj
app  |     flatten(k, v, data)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 578, in _flatten_key_value_pair
app  |     data[k] = self._flatten(v)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 218, in _flatten
app  |     return self._pop(self._flatten_obj(obj))
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 239, in _flatten_obj
app  |     return flatten_func(obj)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 301, in _ref_obj_instance
app  |     return self._flatten_obj_instance(obj)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 474, in _flatten_obj_instance
app  |     return self._flatten_dict_obj(obj.__dict__, data)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 503, in _flatten_dict_obj
app  |     flatten(k, v, data)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 578, in _flatten_key_value_pair
app  |     data[k] = self._flatten(v)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 218, in _flatten
app  |     return self._pop(self._flatten_obj(obj))
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 239, in _flatten_obj
app  |     return flatten_func(obj)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 503, in _flatten_dict_obj
app  |     flatten(k, v, data)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 578, in _flatten_key_value_pair
app  |     data[k] = self._flatten(v)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 218, in _flatten
app  |     return self._pop(self._flatten_obj(obj))
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 239, in _flatten_obj
app  |     return flatten_func(obj)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 242, in _list_recurse
app  |     return [self._flatten(v) for v in obj]
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 242, in <listcomp>
app  |     return [self._flatten(v) for v in obj]
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 218, in _flatten
app  |     return self._pop(self._flatten_obj(obj))
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 239, in _flatten_obj
app  |     return flatten_func(obj)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 301, in _ref_obj_instance
app  |     return self._flatten_obj_instance(obj)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 474, in _flatten_obj_instance
app  |     return self._flatten_dict_obj(obj.__dict__, data)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 503, in _flatten_dict_obj
app  |     flatten(k, v, data)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 578, in _flatten_key_value_pair
app  |     data[k] = self._flatten(v)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 218, in _flatten
app  |     return self._pop(self._flatten_obj(obj))
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 239, in _flatten_obj
app  |     return flatten_func(obj)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 242, in _list_recurse
app  |     return [self._flatten(v) for v in obj]
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 242, in <listcomp>
app  |     return [self._flatten(v) for v in obj]
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 218, in _flatten
app  |     return self._pop(self._flatten_obj(obj))
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 239, in _flatten_obj
app  |     return flatten_func(obj)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 301, in _ref_obj_instance
app  |     return self._flatten_obj_instance(obj)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 474, in _flatten_obj_instance
app  |     return self._flatten_dict_obj(obj.__dict__, data)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 503, in _flatten_dict_obj
app  |     flatten(k, v, data)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 578, in _flatten_key_value_pair
app  |     data[k] = self._flatten(v)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 218, in _flatten
app  |     return self._pop(self._flatten_obj(obj))
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 239, in _flatten_obj
app  |     return flatten_func(obj)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 301, in _ref_obj_instance
app  |     return self._flatten_obj_instance(obj)
app  |   File "/usr/local/lib/python3.6/site-packages/jsonpickle/pickler.py", line 438, in _flatten_obj_instance
app  |     state = obj.__getstate__()
app  |   File "/usr/local/lib/python3.6/site-packages/Cryptodome/PublicKey/RSA.py", line 204, in __getstate__
app  |     raise PicklingError
app  | _pickle.PicklingError

I'm using the jsonpickle library to do the serialization of key-value pairs.

/auth/callback endpoint source code

@app.route("/auth/callback", methods=["GET"])
def auth_callback():
    aresp, atr, idt = consumer.parse_authz(
        query=request.query_string.decode("utf-8"))
    assert aresp["state"] == session["state"]
    consumer.complete(state=aresp["state"])
    flash("You have been logged in")
    return redirect(url_for("index"))

Redis based session storage source code

This piece of art is still work in progress, but the problem relies in the __setitem__ function.

import redis
import jsonpickle
from oic.utils.session_backend import AuthnEvent, SessionBackend
from typing import Any
from typing import Dict
from typing import List
from typing import Optional
from typing import Union
from typing import cast

class RedisSessionBackend(SessionBackend):
    """
    Redis session backend
    """

    def __init__(self, redis_instance: redis.Redis, **args):
        """Create the storage."""
        self.r = redis_instance

    def __setitem__(self, key: str, value: Dict[str, Union[str, bool]]) -> None:
        """Store the session info in the storage."""
        self.r.set(key, jsonpickle.encode(value))

    def __getitem__(self, key: str) -> Dict[str, Union[str, bool]]:
        """Retrieve session information based on session id."""
        data = self.r.get(key)
        return jsonpickle.decode(data.decode("utf-8"))

    def __delitem__(self, key: str) -> None:
        """Delete the session info."""
        self.r.delete(key)

    def __contains__(self, key: str) -> bool:
        return key in self.r.keys()

    def get_by_sub(self, sub: str) -> List[str]:
        """Return session ids based on sub."""
        return [
            key for key in self.r.keys() if jsonpickle.loads(self.r.get(key).decode("utf-8")).get("sub") == sub
        ]

    def get_by_uid(self, uid: str) -> List[str]:
        """Return session ids based on uid."""
        return [
            key
            for key in self.r.keys()
            if AuthnEvent.from_json(self.r.get(key).decode("utf-8"))["authn_event"].uid == uid
        ]

    def get(self, attr: str, val: str) -> List[str]:
        """Return session ids based on attribute name and value."""
        return [
            key for key in self.r.keys() if jsonpickle.loads(self.r.get(key).decode("utf-8")).get(attr) == val
        ]

The custom backend manages to serialize & deserialize some operations perfectly, but values that are highly complex fail to do so. Any clue what could be wrong?

[schlenk]

Seems there is some state in there that cannot be pickled, in your above traceback it is some RSA public key value, probably from fetched jwks of the provider. So this is probably the keystore or some token thats producing trouble.

[alehuo]

Yup, this doesn't happen if I use the basic Python dictionary (that is not persistent and exists in memory). I'd like to have persistent storage for my sessions, to support scaling the application.

Looking from the Cryptodome.PublicKey.RSA, it seems to be not picklable: https://github.com/Legrandin/pycryptodome/blob/128f9bfddd85d32b4ce907caa084e95dd6ae5062/lib/Crypto/PublicKey/RSA.py#L206

I wonder why..

I've opened an issue to the pycryptodome repo about this: https://github.com/Legrandin/pycryptodome/issues/365

[alehuo]

I have solved this issue by creating a custom handler for Cryptodome.Publickey.RSA.

You can find the Redis-based session backend from here: https://github.com/alehuo/pyoidc-redis-session-backend