GoogleCloudPlatform / appengine-python-standard

Google App Engine services SDK for Python 3
Apache License 2.0
70 stars 47 forks source link

Extend / Fix the "NDB_PY2_UNPICKLE_COMPAT" flag #112

Open kaansoral opened 9 months ago

kaansoral commented 9 months ago

'latin1' encoding is the fix to most common fix of the DecodeError's, throughout the docs it's the adopted fix as well, which can be seen here: https://cloud.google.com/appengine/docs/standard/python3/services/access

After this fix, I believe the "NDB_PY2_UNPICKLE_COMPAT" flag should be added to the official docs as well

Fixes #111

google-cla[bot] commented 9 months ago

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

pnico commented 9 months ago

I filed issue #103 asking basically "what is this thing" and got no responses so good luck ever getting this merged :/

If you don't want to use a modified appengine lib, you could try something like this (YMMV, no guarantees)

try:
    _ = self.data
except UnicodeDecodeError:  # we know there's data, probably stored in Python 2
    try:
        prop = self._properties.get('data')
        self.data = pickle.loads(prop._get_base_value(self).b_val, encoding='latin1')
        # we 'fixed' it, we can use self.data now
    except Exception:
        # couldn't fix it, could raise here or just log something scary and give up:
        self.data = None
kaansoral commented 9 months ago

Thank you, this was the response to my second issue tracker issue: https://issuetracker.google.com/issues/322838721

I do share the same sentiment with you, I doubt this will be fixed for others too, I suspect we all push the wagon ourselves at this point those who are left

Have no idea what 'bytes' encoding is for, but I kept it in place so that whatever edge case it fixed, is still fixed