pickle5 problem?

[hvdkamer]

I'm trying to unpack a Ren'Py 8.0.0 archive with the latest version. I get the following error:

Traceback (most recent call last): File "./rpatool", line 363, in <module> archive = RenPyArchive(archive, padlength=padding, key=key, version=version, verbose=arguments.verbose) File "./rpatool", line 65, in __init__ self.load(file) File "./rpatool", line 230, in load self.indexes = self.extract_indexes() File "./rpatool", line 111, in extract_indexes indexes = _unpickle(contents) File "./rpatool", line 24, in _unpickle return pickle.loads(data, encoding='latin1')

System is Debian Buster. I tried to force python3 and installing pickle5 through pip, but that gives the same error. May be I'm doing something wrong? Older archives works flawlessly.

[Triscelion]

Also present in Ren'Py 8.0.1 with MacOS 15.5 and Python 3.10.

[shizmob]

It seems https://github.com/Shizmob/rpatool/pull/13 would fix this, are you able to check?

[schwert]

I had the same problem and just running with python 3.9.13 fixes the pickle 5 problem

On debian the current shebang

#!/usr/bin/env python

Defaults to python 2.7 for me

13 fixes the next problem

[UniverseDevel]

I tested this steps having Python 3.6:

1. Created archive file with RenPy 8+
2. Attempted extract with rpatools resulting in error: ❌

   Traceback (most recent call last):
   File "W:/Projects/Python/RPATool/rpatool.py", line 363, in <module>
   archive = RenPyArchive(archive, padlength=padding, key=key, version=version, verbose=arguments.verbose)
   File "W:/Projects/Python/RPATool/rpatool.py", line 65, in __init__
   self.load(file)
   File "W:/Projects/Python/RPATool/rpatool.py", line 230, in load
   self.indexes = self.extract_indexes()
   File "W:/Projects/Python/RPATool/rpatool.py", line 111, in extract_indexes
   indexes = _unpickle(contents)
   File "W:/Projects/Python/RPATool/rpatool.py", line 24, in _unpickle
   return pickle.loads(data, encoding='latin1')
   ValueError: unsupported pickle protocol: 5

3. Installed Python 3.10 and changed interpreter on project to Python 3.10
4. Attempted extract with rpatools resulting in error Could not extract file file.rpy from archive: 'bytes' object has no attribute 'encode' ❌
5. Applied changes prosposed in #13
6. Attempted extract with rpatools with success ✔️

Make sure you are using correct Python version supporting pickle5 natively as interpreter and #13 should fix this problem.

[hvdkamer]

Thanks for all the work. First I applied the patch. As said above Debian Buster still uses Python 2.7 and that gives the following error:

Traceback (most recent call last): File "../../rpatool", line 366, in <module> archive = RenPyArchive(archive, padlength=padding, key=key, version=version, verbose=arguments.verbose) File "../../rpatool", line 68, in __init__ self.load(file) File "../../rpatool", line 233, in load self.indexes = self.extract_indexes() File "../../rpatool", line 114, in extract_indexes indexes = _unpickle(contents) File "../../rpatool", line 41, in _unpickle return pickle.loads(data) File "/usr/lib/python2.7/pickle.py", line 1388, in loads return Unpickler(file).load() File "/usr/lib/python2.7/pickle.py", line 864, in load dispatch[key](self) File "/usr/lib/python2.7/pickle.py", line 892, in load_proto raise ValueError, "unsupported pickle protocol: %d" % proto ValueError: unsupported pickle protocol: 5

I then changed the first line to python3 which is version 3.7 and that gives:

Traceback (most recent call last): File "../../rpatool", line 366, in <module> archive = RenPyArchive(archive, padlength=padding, key=key, version=version, verbose=arguments.verbose) File "../../rpatool", line 68, in __init__ self.load(file) File "../../rpatool", line 233, in load self.indexes = self.extract_indexes() File "../../rpatool", line 114, in extract_indexes indexes = _unpickle(contents) File "../../rpatool", line 27, in _unpickle return pickle.loads(data, encoding='latin1') ValueError: unsupported pickle protocol: 5

I then installed through pip3 the packages cloudpickle and pickle5 as suggested elsewhere on the internet. This gives still the same error. I then changed the import line as follows:

import pickle5 as pickle

This is suggested here. That works like a charm. So the patch does it job, but I think there is more to be added. This newer archive is not compatible with Python 2.7 where the older archives work. So there should be some kind of test there? And also an check for importing the newer pickle5?

[schwert]

I hadn't realized pickle 5 was a 3.8 thing.

@Shizmob do you want me to amend my PR to add a check on the python version and import that module if needed?

[hvdkamer]

The pickle5 backport says it works from Python 3.5 to 3.7 because from 3.8.3 it is standard. I can confirm this backport works for me in Python 3.7. However it also means that older versions can not extract the newer archives.

[Triscelion]

13 fixes it with later python 3s (3.9, 3.10) as far as I can see.

[JamiesonC]

I can confirm that #13 fixes the 'bytes' object has no attribute 'encode' error that occurs under Python 3.8.10.

[ProtoFoo]

I can also confirm that I was perfectly able to unpack multiple Ren'Py 8 archives with the change in https://github.com/Shizmob/rpatool/pull/13 I'm using Python 3.10.5

[shizmob]

I just pushed a change that will attempt to use pickle5 if available on older Python versions, and warn otherwise. I hope together with the merge of #13 the issue is now solved as much as possible!