Key Error with PickleProperty

[weeklymize]

In one model I use a PickleProperty to save dictionary or array values.

Unfortunately I get this Error, when posting to the endpoint: Encountered unexpected error from ProtoRPC method implementation: KeyError ('\xb5')

I tried it with a dictionary, array and string object.

[dhermes]

@weeklymize can you post some code? It seems like you dictionary is just missing a key, but it also seems like that key could come from a serialized bytestring.

[weeklymize]

Sure.

That's the model:

class Module(EndpointsModel):
    module_type = ndb.StringProperty(required=True)
    slot = ndb.IntegerProperty(required=True)
    details = ndb.PickleProperty()

which is used as StructuredProperty for this Model:

class Report(EndpointsModel):
    name = ndb.StringProperty(required=True)
    modules = ndb.StructuredProperty(Module, repeated=True)

[dhermes]

Note that PickleProperty inherits from ndb.BlobProperty so this should map to messages.BytesField.

This means in order to use it you'll need to manually pass base64 encoded byte-strings to the API. This is likely not what you want.

Can you also provide an API definition and an example call that fails? To show a call, feel free to take a screenshot of localhost:8080/_ah/api/explorer in use.

[weeklymize]

Ah, ok. I understand.

This a request:

POST http://localhost:9080/_ah/api/sbapi/v1/report

Content-Type:  application/json
X-JavaScript-User-Agent:  Google APIs Explorer

{
 "account": "8239472394",
 "compare": true,
 "name": "Testreport",
 "modules": [
  {
   "details": "{'data':'test'}",
   "module_type": "twitter_report"
  }
 ]
}

Response:
503 Service Unavailable

Headers

{
 "error": {
  "code": 503,
  "errors": [
   {
    "domain": "global",
    "message": "Internal Server Error",
    "reason": "backendError"
   }
  ],
  "message": "Internal Server Error"
 }
}

Log:

Traceback (most recent call last):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/protorpc-1.0/protorpc/wsgi/service.py", line 181, in protorpc_service_app
    response = method(instance, request)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/endpoints-1.0/endpoints/api_config.py", line 1332, in invoke_remote
    return remote_method(service_instance, request)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/protorpc-1.0/protorpc/remote.py", line 412, in invoke_remote_method
    response = method(service_instance, request)
  File "lib/endpoints_proto_datastore/ndb/model.py", line 1416, in EntityToRequestMethod
    request = cls.FromMessage(request)
  File "lib/endpoints_proto_datastore/ndb/model.py", line 1238, in FromMessage
    to_add = [FromValue(value_property, element) for element in value]
  File "lib/endpoints_proto_datastore/ndb/model.py", line 145, in FromValue
    return modelclass.FromMessage(value)
  File "lib/endpoints_proto_datastore/ndb/model.py", line 1240, in FromMessage
    to_add = FromValue(value_property, value)
  File "lib/endpoints_proto_datastore/ndb/model.py", line 152, in FromValue
    return pickle.loads(value)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
    return Unpickler(file).load()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
KeyError: '~'

[dhermes]

Thanks. I was able to recreate and get a similar failure: https://gist.github.com/dhermes/9a5ecf4579489c6cb6f4

I am going to dig a bit deeper but overall I recommend against using a PickleProperty. There is almost always a reason to use actual properties (i.e. so you can create filters). Also just asking for a blob of text such as

"{'data':'test'}"

is not so useful.

[dhermes]

@weeklymize Thanks for filing this. I think this is a legitimate bug, and I opened #115 to fix it.

OTOH I still strongly recommend against using a PickleProperty.

[weeklymize]

Thanks. I will rethink the structure as well.