search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.33k stars 453 forks source link

p_group_cohomology does not build on Python3 #28414

Closed dimpase closed 4 years ago

dimpase commented 5 years ago

The aim of this ticket is to provide a new version of the optional p_group_cohomology spkg so that it builds and passes tests both with python-2 and python-3.

Hopefully the version number 3.3 (two times three) is appropriate.

Package documentation

Source ball

p_group_cohomology at Travis CI

Depends on #28444

CC: @simon-king-jena

Component: packages: optional

Author: Simon King

Branch: 6c5dafc

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/28414

jhpalmieri commented 5 years ago
comment:1

cmp is also not supported in Python 3, and for a dictionary d, you should do x in d rather than d.has_key(x).

jhpalmieri commented 5 years ago
comment:2

Simon, I have made some progress in converting this to Python 3. Let me know and I can email you some diff files.

simon-king-jena commented 5 years ago
comment:3

Hi John, in fact Python-3-compatibility was one of the things that I considered for the next package version. So, if you already have a diff, I'd appreciate to see it!

jhpalmieri commented 5 years ago
comment:4

I just sent email to your Jena address. I can get the package to build, but it doesn't work yet. An illustration of one problem:

sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/src/bin/sage-ipython:1: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  #!/usr/bin/env sage-python23
WARNING: Files on disk have been moved or are not writeable.
> Will try to recover later.
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py:623: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  has_rich_repr = isinstance(obj, SageObject) and hasattr(obj, '_rich_repr_')
8gp3: Files on disk have been moved - trying to get things right
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-5-9bd7fbf5ad54> in <module>()
----> 1 load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/IPython/core/displayhook.py in __call__(self, result)
    244             self.start_displayhook()
    245             self.write_output_prompt()
--> 246             format_dict, md_dict = self.compute_format_data(result)
    247             self.update_user_ns(result)
    248             self.fill_exec_result(result)

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/IPython/core/displayhook.py in compute_format_data(self, result)
    148 
    149         """
--> 150         return self.shell.display_formatter.format(result)
    151 
    152     # This can be set to True by the write_output_prompt method in a subclass

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/display/formatter.py in format(self, obj, include, exclude)
    198         """
    199         # First, use Sage rich output if there is any
--> 200         sage_format, sage_metadata = self.dm.displayhook(obj)
    201         assert PLAIN_TEXT in sage_format, 'plain text is always present'
    202         if not set(sage_format.keys()).issubset(self.default_mime()):

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py in displayhook(self, obj)
    809             return
    810         self._backend.set_underscore_variable(obj)
--> 811         plain_text, rich_output = self._rich_output_formatter(obj, dict())
    812         return self._backend.displayhook(plain_text, rich_output)
    813 

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py in _rich_output_formatter(self, obj, rich_repr_kwds)
    621         rich_output = None
    622         plain_text = None
--> 623         has_rich_repr = isinstance(obj, SageObject) and hasattr(obj, '_rich_repr_')
    624         if has_rich_repr:
    625             rich_output = self._call_rich_repr(obj, rich_repr_kwds)

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getattr__()

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
    149 
    150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
    152         X = loads(fobj.read(), compress=compress)
    153     try:

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
    150     ## Load file by absolute filename
    151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
    153     try:
    154         X._default_filename = os.path.abspath(filename)

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    967 
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 
    971 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
simon-king-jena commented 5 years ago
comment:5

Replying to @jhpalmieri:

I just sent email to your Jena address. I can get the package to build, but it doesn't work yet.

Thank you very much, I received the patch.

An illustration of one problem:

sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/src/bin/sage-ipython:1: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

That probably is harmless. I.e., I should use "warning", and apparently in some unpickling method I use "str" where it should be "bytes".

jhpalmieri commented 5 years ago
comment:6

Yes, and I would like to track down the str vs. bytes issue. It would be nice if I could get basic functionality like CohomologyRing(8,3) to work.

jhpalmieri commented 5 years ago
comment:7

Changing logger.warn (or coho_logger.warn) to X.warning fixes the deprecation message. I don't know about the str/bytes issue.

simon-king-jena commented 5 years ago
comment:8

Replying to @jhpalmieri:

Changing logger.warn (or coho_logger.warn) to X.warning fixes the deprecation message. I don't know about the str/bytes issue.

I did change .warn( into .warning( everywhere (so, no need to include it in a diff).

Concerning str vs. byte: There is only one occurrence of cdef str, and this gets assigned to the output of <some singular object>.name(). Is that output really a string in python-3? If not, it should be changed to cdef bytes. However, so far the error in unpickling stuff looks mysterious to me, as the traceback doesn't mention where the problem actually occurs in my code.

jhpalmieri commented 5 years ago
comment:9

Could the problem be in the way the .sobj files were produced? What if you regenerate them using Python 3?

simon-king-jena commented 5 years ago
comment:10

Replying to @jhpalmieri:

Could the problem be in the way the .sobj files were produced? What if you regenerate them using Python 3?

You mean Python 3 is severely backwards incompatible and cannot deal with old pickles? That would be a show stopper for Sage, I think.

Note that at some point in the past I inserted code to regenerate very old pickles and translate them automatically to a new format. However, I would certainly not like to regenerate several gigabytes of data in the web repository.

Note that the problem occurs together with a warning:

WARNING: Files on disk have been moved or are not writeable.
> Will try to recover later.
...
8gp3: Files on disk have been moved - trying to get things right

Maybe that indicates where the problem comes from. So, let's recall how storing the modular cohomology ring of a prime power group is solved.

Very early in the package development I found that is is terribly slow to pack all data of a ring in a single file. It simply takes so long and requires so much memory (at least in some cases) that it is not feasible to automatically store intermediate results of a lengthy computation (which evidently is a reasonable thing to do).

My solution was to not store a ring in a single file but distribute the data in a folder. So, the actual .sobj pickle file only contains basic information but is not stand-alone. In particular, the p_group_cohomology spkg needs to know where the data folder is located, the .sobj file has to be in the data folder and also it needs write permission for otherwise it couldn't do a continuation of the cohomology ring's computation. What is stored in the web repository is a compressed tar file of the data folder (i.e., when the web repository is accessed, then the tar file is downloaded, unpacked, and then the cohomology ring is loaded from there).

By default, the data folder's location is composed from the location of the current workspace (which are global variables) and from the identifier of the group (which is stored in the .sobj file).

The warning mentioned above occurs when the data folder's default location cannot be accessed. And in that case, I do some trickery --- and perhaps that trickery doesn't work in Python 3? Namely:

If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.

simon-king-jena commented 5 years ago
comment:11

Question: How to test whether something has a type that qualifies it as a filename? Is isinstance(self._property_dict['_need_new_root'],basestring) ok in Python-3?

simon-king-jena commented 5 years ago
comment:12

Replying to @simon-king-jena:

Question: How to test whether something has a type that qualifies it as a filename? Is isinstance(self._property_dict['_need_new_root'],basestring) ok in Python-3?

Here is what I found in the "internetz" concerning how to test if something is a string in a way that works in python 2 and 3:

import six

if isinstance(obj, six.string_types):
    print('obj is a string!')

So, I think I should fix this.

simon-king-jena commented 5 years ago
comment:13

Replying to @simon-king-jena:

If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.

Hooray:

# in Sage-with-python-2
sage: P = CombinatorialFreeModule(ZZ,'x')
sage: save(P,'../../pickle')
# BOTH with python-2 and python-3
sage: B = load('../../pickle.sobj')
sage: B._default_filename
'/home/king/Sage/pickle.sobj'

So, the old trick should still work.

simon-king-jena commented 5 years ago
comment:14

Currently, I get the following behaviour:

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.set_workspace(tmp_dir())
sage: H = CohomologyRing(8,3,from_scratch=True,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/'
Computing basic setup for Small Group number 1 of order 2
Computing basic setup for Small Group number 2 of order 4
Computing basic setup for Small Group number 3 of order 8
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-d1c5c11acd7a> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
   1246                 OUT = COHO(gap(KEY[0]), **extras)
   1247         else:
-> 1248             OUT = COHO(KEY[0],KEY[1], **extras)
   1249         _gap_reset_random_seed()
   1250         try:

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__init__()

TypeError: expected bytes, str found

and

sage: H = CohomologyRing(8,3,options='debug')
Local data found at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj
Creating symbolic links from /home/king/.sage/temp/klap/26693/dir_f6_f2ei3/8gp3 to /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3
Group data are rooted at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/'
The state descriptor of the to-be-unpickled ring is expected to be provided at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/8gp3/dat/State.sobj'
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds)
   1154                 coho_options['@use_this_root@'] = root_workspace
-> 1155                 OUT = load(os.path.join(root_workspace,file_name)) # realpath here?
   1156                 if '@use_this_root@' in coho_options:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
    150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
    152         X = loads(fobj.read(), compress=compress)

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
    151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
    153     try:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO_unpickle()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
    150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
    152         X = loads(fobj.read(), compress=compress)

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
    151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
    153     try:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-4-927128be23f4> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),options='debug')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds)
   1159                 if '@use_this_root@' in coho_options:
   1160                     del coho_options['@use_this_root@']
-> 1161                 raise IOError("Saved data at %s are not readable: %s"%(os.path.join(root_local_sources,file_name), msg))
   1162         ## 4. Search web repository
   1163         elif kwds.get('websource')!=False and (not from_scratch):

OSError: Saved data at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj are not readable: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

So, that's something I can now try to debug.

fchapoton commented 5 years ago
comment:15

in case you need them, sage has its own compatibility functions "bytes_to_str" and "str_to_bytes", to be found in sage.cpython.string and working in both py2 and py3

simon-king-jena commented 5 years ago
comment:16

Replying to @fchapoton:

in case you need them, sage has its own compatibility functions "bytes_to_str" and "str_to_bytes", to be found in sage.cpython.string and working in both py2 and py3

Thank you! I don't know yet whether or not I'll need them, but certainly it is good to know.

simon-king-jena commented 5 years ago
comment:17

I found that this fails in RESL.__init__:

        tmprstem = os.path.join(res_folder,rstem)
        tmpgstem = os.path.join(gps_folder,gstem)
        self.Data = newResolWithGroupLoaded(tmprstem,tmpgstem,1)

Apparently tmprstem or tmpgstem are strings, but the C function newResolWithGroupLoaded expects char * --- is there no automatic conversion from py3-string to char*?

fchapoton commented 5 years ago
comment:18

Maybe try

self.Data = newResolWithGroupLoaded(str_to_bytes(tmprstem),str_to_bytes(tmpgstem),1)

..with no guarantee..

simon-king-jena commented 5 years ago
comment:19

Replying to @fchapoton:

Maybe try

self.Data = newResolWithGroupLoaded(str_to_bytes(tmprstem),str_to_bytes(tmpgstem),1)

..with no guarantee..

That is likely to work. Another potential solution would be to generally work with bytes. I.e., let self.gstem be a bytes and then do os.path.join(self.res_folder, b'A'+self.gstem+b'.sobj')) etc.

What would be the preferred way?

fchapoton commented 5 years ago
comment:20

beware that you cannot concatenate bytes and str, so everything must become bytes. Not the way to go in my opinion.. moreover, not sure that os.* can handle bytes

EDIT:

indeed:

sage: os.path.join('home', b'jojo')
...
TypeError: Can't mix strings and bytes in path components
simon-king-jena commented 5 years ago
comment:21

Replying to @fchapoton:

beware that you cannot concatenate bytes and str, so everything must become bytes. Not the way to go in my opinion.. moreover, not sure that os.* can handle bytes

EDIT:

indeed:

sage: os.path.join('home', b'jojo')
...
TypeError: Can't mix strings and bytes in path components

This only shows that it cannot mix both types. In fact, if both are bytes then things work:

sage: os.path.join(b'home', b'foo')
b'home/foo'

Probably it would be a long and tedious work to change everything to bytes. In principle I would be willing to do that work, but if there is a different solution involving less work and only a slight overhead, I'd prefer that.

Currently, using str_to_bytes when passing a string to a C function does seem to work.

simon-king-jena commented 5 years ago
comment:22

WTF? I just see that I have the following argument types of COCH.__init__:

def __init__(self, PARENT, int n, char *Nick, L, ydeg=None, rdeg=None, is_polyrep=False)

I suppose Nick shouldn't be char* but (if anything) str. Or better not type it at all.

simon-king-jena commented 5 years ago
comment:23

I think I should also fix the following compiler warning (there are several similar):

pGroupCohomology/cohomology.c: In function ‘__pyx_pf_16pGroupCohomology_10cohomology_12explore_one_parameter’:
pGroupCohomology/cohomology.c:17678:35: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     for (__pyx_t_7 = 0; __pyx_t_7 < __pyx_t_16; __pyx_t_7+=1) {
simon-king-jena commented 5 years ago
comment:24

I guesss getting the following error is a progress, although it seems a bit strange to me:

sage: H = CohomologyRing(8,3,from_scratch=True,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/31253/dir_88iattmw/'
Computing basic setup for Small Group number 1 of order 2
Computing basic setup for Small Group number 2 of order 4
Computing basic setup for Small Group number 3 of order 8
> export action matrices
Initialising maximal p-elementary abelian subgroups
Inserting SmallGroup(2,1) as a subgroup
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/31253/dir_88iattmw/'
> export action matrices
Unable to save basic ring setup
Traceback (most recent call last):
  File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)
    obj.save(filename=filename, compress=compress, **kwargs)
AttributeError: 'tuple' object has no attribute 'save'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3636)
    obj.save(filename=filename, compress=compress, **kwargs)
  File "sage/structure/sage_object.pyx", line 424, in sage.structure.sage_object.SageObject.save (build/cythonized/sage/structure/sage_object.c:3328)
    filename = _base_save(self, filename, compress=compress)
  File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
    with open(filename, 'wb') as fobj:
  File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
    fobj.write(_base_dumps(obj, compress=compress))
  File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
    gherkin = SagePickler.dumps(obj)
  File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
    pickler.dump(obj)
  File "pGroupCohomology/cohomology.pyx", line 3464, in pGroupCohomology.cohomology.COHO.__reduce__
  File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py", line 85, in safe_save
    save (obj, path)
  File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)
    _base_save(obj, filename, compress=compress)
  File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
    with open(filename, 'wb') as fobj:
  File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
    fobj.write(_base_dumps(obj, compress=compress))
  File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
    gherkin = SagePickler.dumps(obj)
  File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
    pickler.dump(obj)
TypeError: can't pickle dict_items objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)
    obj.save(filename=filename, compress=compress, **kwargs)
AttributeError: 'tuple' object has no attribute 'save'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py", line 1254, in _get_p_group_from_scratch
    safe_save(OUT, OUT.autosave_name())
  File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py", line 85, in safe_save
    save (obj, path)
  File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)
    _base_save(obj, filename, compress=compress)
  File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
    with open(filename, 'wb') as fobj:
  File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
    fobj.write(_base_dumps(obj, compress=compress))
  File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
    gherkin = SagePickler.dumps(obj)
  File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
    pickler.dump(obj)
  File "pGroupCohomology/cohomology.pyx", line 3464, in pGroupCohomology.cohomology.COHO.__reduce__
  File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py", line 85, in safe_save
    save (obj, path)
  File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)
    _base_save(obj, filename, compress=compress)
  File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
    with open(filename, 'wb') as fobj:
  File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
    fobj.write(_base_dumps(obj, compress=compress))
  File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
    gherkin = SagePickler.dumps(obj)
  File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
    pickler.dump(obj)
TypeError: can't pickle dict_items objects
Computing next term
> rk P_02 =   1
We have to choose 1 new generator in degree 1
> There is 1 Duflot regular generator in degree 1
Summary: 0 relations and 1 generators in degree 1
Ring approximation computed out to degree 1!
Storing approximation data
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

AttributeError: 'pGroupCohomology.resolution.RESL' object has no attribute 'save'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

AttributeError: 'dict_items' object has no attribute 'save'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3636)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/structure/sage_object.pyx in sage.structure.sage_object.SageObject.save (build/cythonized/sage/structure/sage_object.c:3328)()
    423 
--> 424         filename = _base_save(self, filename, compress=compress)
    425 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__reduce__()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getstate__()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     84         os.unlink(path)
---> 85     save (obj, path)
     86 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__reduce__()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.exportLifts()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.LIFTcontainer.export()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     84         os.unlink(path)
---> 85     save (obj, path)
     86 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()

TypeError: can't pickle dict_items objects

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

AttributeError: 'pGroupCohomology.resolution.RESL' object has no attribute 'save'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

AttributeError: 'dict_items' object has no attribute 'save'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-3-d1c5c11acd7a> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
   1246                 OUT = COHO(gap(KEY[0]), **extras)
   1247         else:
-> 1248             OUT = COHO(KEY[0],KEY[1], **extras)
   1249         _gap_reset_random_seed()
   1250         try:

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InitSubgroups()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InsertSubgroup()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.make()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.next()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     83     if os.path.islink(path):
     84         os.unlink(path)
---> 85     save (obj, path)
     86 
     87 ###################################

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:
    244         # Saving an object to an image file.

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    177     filename = _normalize_filename(filename)
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    178 
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 
    182     return filename

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    255     """
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 
    259     if compress:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    834         buf = io.BytesIO()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()
    838 

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__reduce__()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getstate__()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     83     if os.path.islink(path):
     84         os.unlink(path)
---> 85     save (obj, path)
     86 
     87 ###################################

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:
    244         # Saving an object to an image file.

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    177     filename = _normalize_filename(filename)
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    178 
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 
    182     return filename

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    255     """
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 
    259     if compress:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    834         buf = io.BytesIO()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()
    838 

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__reduce__()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.exportLifts()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.LIFTcontainer.export()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     83     if os.path.islink(path):
     84         os.unlink(path)
---> 85     save (obj, path)
     86 
     87 ###################################

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:
    244         # Saving an object to an image file.

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    177     filename = _normalize_filename(filename)
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    178 
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 
    182     return filename

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    255     """
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 
    259     if compress:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    834         buf = io.BytesIO()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()
    838 

TypeError: can't pickle dict_items objects

What are dict_items, and why can't they be pickled?

fchapoton commented 5 years ago
comment:25

If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).

You may need to wrap with list(d.items()) for pickling.

simon-king-jena commented 5 years ago
comment:26

Replying to @fchapoton:

If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).

You may need to wrap with list(d.items()) for pickling.

Wouldn't that considerably slow things down, at least in Python-2? After all, d.items() is a list, and list(...) applied to a list creates a (shallow) copy of that list.

simon-king-jena commented 5 years ago
comment:27

There is a considerable progress now (after putting list around .items()):

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.set_workspace(tmp_dir())
sage: H = CohomologyRing(8,3,from_scratch=True,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/'
Computing basic setup for Small Group number 1 of order 2
Computing basic setup for Small Group number 2 of order 4
Computing basic setup for Small Group number 3 of order 8
> export action matrices
Initialising maximal p-elementary abelian subgroups
Inserting SmallGroup(2,1) as a subgroup
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/'
> export action matrices
Computing next term
> rk P_02 =   1
We have to choose 1 new generator in degree 1
> There is 1 Duflot regular generator in degree 1
Summary: 0 relations and 1 generators in degree 1
Ring approximation computed out to degree 1!
Storing approximation data
Determine degree 2 standard monomials
We got 1 standard monomials
Monomial 0
> Candidate: c_1_0^2
> Express monomial as a Cochain
Compute c_1_0*c_1_0
Compose chain maps R_2 -> R_1 -> R_0
Lift with Urbild Groebner basis in degree 1
load Urbild Groebner basis
Decomposable cochain found

There is no new generator in degree 2
Summary: 0 relations and 0 generators in degree 2
Ring approximation computed out to degree 2!
Storing approximation data
Inserting SmallGroup(4,2) as a subgroup
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/'
> export action matrices
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-3-d1c5c11acd7a> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
   1246                 OUT = COHO(gap(KEY[0]), **extras)
   1247         else:
-> 1248             OUT = COHO(KEY[0],KEY[1], **extras)
   1249         _gap_reset_random_seed()
   1250         try:

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InitSubgroups()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InsertSubgroup()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
   1246                 OUT = COHO(gap(KEY[0]), **extras)
   1247         else:
-> 1248             OUT = COHO(KEY[0],KEY[1], **extras)
   1249         _gap_reset_random_seed()
   1250         try:

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

KeyError: 'autoliftElAb'

The good news is that for the first time cohomology rings were computed in py-3 (namely cohomology rings of elementary abelian groups).

jhpalmieri commented 5 years ago
comment:28

Replying to @simon-king-jena:

Replying to @fchapoton:

If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).

You may need to wrap with list(d.items()) for pickling.

Wouldn't that considerably slow things down, at least in Python-2? After all, d.items() is a list, and list(...) applied to a list creates a (shallow) copy of that list.

You could use a try ... except block to deal with the Python 2 case quickly. Sage will soon (I hope) default to Python 3, so I don't think this is worth it, though.

simon-king-jena commented 5 years ago
comment:29

Replying to @simon-king-jena:

There is a considerable progress now (after putting list around .items()): ...

KeyError: 'autoliftElAb'

Apparently I forgot to create a shallow copy in some places in which I temporarily stored the global options -- it doesn't work to recreate a dict from its items in Py-3, and that's why I got the key error above:

sage: D = {1:2}
sage: I = D.items()
sage: D.clear()
sage: D.update(I)
sage: D
{}

In Py-2, one would get

sage: D
{1: 2}
simon-king-jena commented 5 years ago
comment:30

Hooray!

I am now in a position to completely compute CohomologyRing(8,3) from scratch. Unfortunately reading old data doesn't work yet, but certainly it is a progress :-)

jdemeyer commented 5 years ago
comment:31

Replying to @simon-king-jena:

import six

Please don't do this in Cython files. There is almost never a reason to use six in Cython since Cython handles most of the Python 2/3 incompatibilities for you.

jdemeyer commented 5 years ago
comment:32

Replying to @simon-king-jena:

Replying to @fchapoton:

If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).

You may need to wrap with list(d.items()) for pickling.

Wouldn't that considerably slow things down, at least in Python-2?

Use list(d.iteritems()) in Cython.

simon-king-jena commented 5 years ago
comment:33

Replying to @jdemeyer:

Replying to @simon-king-jena:

import six

Please don't do this in Cython files. There is almost never a reason to use six in Cython since Cython handles most of the Python 2/3 incompatibilities for you.

I am not dealing with incompatibilities in that case. I import six in order to be able to safely test for string types. How to, if I don't use six?

simon-king-jena commented 5 years ago
comment:34

Replying to @jdemeyer:

Use list(d.iteritems()) in Cython.

Would this also work fast if Cython doesn't know that d is a dict? Should I do list(<dict>(self._D).iteritems()), for instance?

jdemeyer commented 5 years ago
comment:35

Replying to @simon-king-jena:

I import six in order to be able to safely test for string types. How to, if I don't use six?

First of all, you should check whether you really need to test for "string types". Can you use duck typing instead? If for some reason, duck typing is hard, then it depends what you mean precisely with "string types".

Since you were mentioning filenames: isinstance(s, (bytes, unicode)) are valid types for filenames both in Python 2 and Python 3. Note that Python 3 also has https://docs.python.org/3/library/pathlib.html, a custom class to deal specifically with filenames.

simon-king-jena commented 5 years ago

Pickle that isn't readable in Python-3

simon-king-jena commented 5 years ago
comment:36

Attachment: State.sobj.gz

Replying to @jhpalmieri:

I can get the package to build, but it doesn't work yet. An illustration of one problem:

sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')
... 
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

I tried to track down that problem, and currently it seems to me that it is due to a very bad backwards incompatibility in Python. You can reproduce it as follows:

Now, in Sage-with-Py-2, loading the file works:

sage: X = load('/home/king/Projekte/coho/tests/State.sobj')
sage: X
([[1,
   [(2, 1),
    [
[1 0 0 0 0 0 0 0]                                                                                                                   
[0 0 0 1 1 1 1 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg1_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg1_2']
]]],
  [2,
   [(4, 2),
    [
[1 0 0 0 0 0 0 0]                                                                                                                   
[0 0 0 1 1 1 1 1]                                                                                                                   
[0 1 0 0 0 0 0 0]                                                                                                                   
[0 0 0 0 0 1 0 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg2_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg2_2']
]]],
  [3,
   [(4, 2),
    [
[1 0 0 0 0 0 0 0]                                                                                                                   
[0 0 0 1 1 1 1 1]                                                                                                                   
[0 0 1 0 0 0 0 0]                                                                                                                   
[0 0 0 0 0 0 1 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg3_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg3_2']
]]]],
 [2, 1, 1],
 1,
 1,
 '8gp3/',
 ['b_1_0*b_1_1'],
 '8gp3/dat/',
 [((4, 2), '/home/king/.sage/pGroupCohomology/db/4gp2/H4gp2'),
  ((2, 1), '/home/king/.sage/pGroupCohomology/db/2gp1/H2gp1')],
 '8gp3/dat/',
 [],
 '8gp3/sgp/',
 2,
 [2, 3],
 [2, 2],
 2,
 2,
 ['b_1_0*b_1_1'],
 [[2, 'c_2_2', [0 0 1], 1, 0],
  [1, 'b_1_0', [1 0], 0, 0],
  [1, 'b_1_1', [0 1], 0, 0]],
 [[0, [['1', ['1']]]],
  [1, [['b_1_0', ['b_1_0']], ['b_1_1', ['b_1_1']]]],
  [2,
   [['c_2_2', ['c_2_2']],
    ['b_1_0', ['b_1_0^2']],
    ['b_1_1', ['b_1_0*b_1_1', 'b_1_1^2']]]]],
 {},
 0.24,
 True,
 'Monomials',
 2,
 None,
 '',
 0,
 '8gp3',
 '8gp3/dat/R8gp3.sobj',
 1,
 ['0'],
 ['b_1_0 + b_1_1'],
 None,
 [('_max_module_deg', 0),
  ('sgpDickson', [((4, 2), [[1, 'c_1_1', [0 1]]])]),
  ('useFactorization', True),
  ('useElimination', False),
  ('GroupDescr', 'Dihedral group of order 8'),
  ('_key', ((8, 3), '/home/king/SPKG/database/8gp3/dat/State')),
  ('auto', 4),
  ('Restriction_2', ['c_1_0*c_1_1+c_1_0^2', 'c_1_1', '0']),
  ('GroupName', 'D8'),
  ('DicksonExp', 3),
  ('_parameters_do_exist', True),
  ('KeepBases', None),
  ('completeGroebner', True),
  ('_SymondsTestdata', ['c_2_2', 'b_1_1', 'b_1_0']),
  ('_parameters_for_criterion', ['c_2_2', 'b_1_1', 'b_1_0']),
  ('root', '/home/king/SPKG/database/'),
  ('Restriction_3', ['c_1_0*c_1_1+c_1_0^2', '0', 'c_1_1']),
  ('_method', 'Symonds')],
 [(('dependent_parameters',), [['c_2_2', 'b_1_1', 'b_1_0'], 2])])

However, Sage-with-Py-3 yields the error found by John:

sage: X = load('/home/king/Projekte/coho/tests/State.sobj')
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-4-74e7e91f9969> in <module>()
----> 1 X = load('/home/king/Projekte/coho/tests/State.sobj')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
    149 
    150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
    152         X = loads(fobj.read(), compress=compress)
    153     try:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
    150     ## Load file by absolute filename
    151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
    153     try:
    154         X._default_filename = os.path.abspath(filename)

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    967 
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 
    971 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

And the point is: Python-3 even is unable to read the pickle file as a string! In Python-2, reading the file is no problem at all.

king@klap:~$ ~/Sage/git/py3/sage -python
Python 3.7.3 (default, Aug 27 2019, 23:22:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('/home/king/Projekte/coho/tests/State.sobj')
>>> f.read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/king/Sage/git/py3/local/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 1: invalid start byte
>>> 
king@klap:~$ ~/Sage/git/sage/sage --python
Python 2.7.15 (default, Jul 26 2019, 11:49:43) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('/home/king/Projekte/coho/tests/State.sobj')
>>> f.read()
'x\x9c\x95U\x8bZ\x1bE\x14&!\x91\x92J\x8b\x82V\xbc\x94x\xedXl\x96\x04\xaah\x95B\x0b\x05]\xc1\x16\x18\xad\x97\xe9:\xd9\x1dv\xb7d3{v7\nU,^\xd8B\xbd\xd6\xfb\xfd\xae\xcf\xd0\xa7\xf0E|\t\xcfl\xb2\x08\xf5\xab_I\xbe\x9d\xefd\xe6?\xe7\xfc\xe7?g6kY\xc2 \x83O\x96\xe8\x19\x06\xed\xc4\x0c\xb9-J\x81[\xb7\xc3\x92[\x8f\x84-\x82\x82\xc7\x97\x84\x91\xfe\x80\x1c\xcdT\xd6\xe7 \xef\xa0QF\xe3\xa6\x18:\x18\xec!\xc4\xf4\xa7\x02\xd9\xf0OJGz\xb2&\xed\x95\x92\x17-\x17f\x16\xce\x1a\x8d\xba\xef\x9aK5a\x985\x1e\x86\x05\xe8\x94PX\x85\xbd\xf4\x80a\x84|Q\x18\x8b2H1\x98\xd80\xe0\xe6\x8d\xb0J\xf4,~\xf7\xd0\xecZ\xffF4\x07]\x0c\xf6\x11\xa7\xa0v3\xb8\x9bY\xdb\xc4\xcd\xfdt@\xc3lB[B?m\xfe\xb4>\xa5Y<\xe2U\x1e\nm\xc4\xf6\x87\xb4\xd0\xf6\x13#\xb4\xcbF\x19\xbaw\x03\xaf\xc0-\x02?\x0cn\xc5\x9c\x0cz\x88*x\x18\x0b\xeeuZ\x12\xdc\x16\xc3\xed\x0c\x0e\xb4X\xe5\x90Un\xad\x7f,\xaf\xd8\xde\xc1\xa0\xef?l\xef\xbc\xf1\xf4\x15d{\xd7n\xe0\x15\xb8\xbb\xc9\xf6\x1e\xa2\xb738\xb8\xc5\xb6?e[\x8c\xe1^\x06\xf7\xedd[lWl\xefg\xf0\x00qvAo\x08\xe9=\xb8\x1bx\x05\x1e\x12M~\x87\x9a\x9adD:?$5\x1e\xa6\xf9\xc4\x8f\xc1a\xba\xb7j\x94\x8d\xc1\xc3j-\xc3\x00\xa7\x9d\xc9\t\xc6\xc7\xd3G\x88\xd3\xeb\xa0\xf6G\xa8\xb6\x8d@I\r\xaev\xed\x08jVU\x1b\xb6\xfd\x8a6\xad\xd6\x18JN\xde\xc1\x81\xd5n\xd0\xb5b\xfbemZ\xad1\x0c\x8a\xed,Z\xb6\xaaQMG\x99\xa42W\x941\x84\xc6\x10\x16;\xbc\xb5}45\x1e\x15\xceQ\xe5\xf1\xd8\xce\x1aG8\x83\xc7\xf1\x1e>\x81\xf2\xd0\xbci\xa8\x8e\x1eK\x07\xa8\x9df\x8aj\x80\x9eD\xbb\r\xc3>\x85\x97\x95\xe6\x13w\x18MA\xd9\xd6\x94\x1d\xd7\xdb\x12\xd0\xd8\x16\xa8\x0c\xe3\xdb@c\nt"\x01!\xea$\xe6\x9c z\x1b\x83I\x06\xa7\x08v\x82\xc1\x14\xae\\p<\x9dN\xde\nO#\xe6\x19\xe2\x8c2\xd0\xd3\xac\xcf\xaa\xd3\x19\xe2\x8c3\x98M\x93<\xc7\x93\x06\x9fN\xae\xcb\x19\xf4\x99#\xce1\x06\xf3i9\x0b\xca\x87\x924\xc4\xf3\x0c^\xa0\x1d\x89}\xae\x02g\xd5\xe1\x8b$\x8d\xf5\x12\x83\x97\xc9N\x85^i\x82\xcb\x08f\xc9,\xad\xc2\xb9\xa9\xe3\x7f]=\xf3\xf7\xfa\xc1\xab\x1b\xb4sF\xd6\xa5\xe7\xf2Z\x08F\xaa\xf5\xab\xb3\x14\xcb\xa49\xd5*\xe0\xb4g\xab}s\xca*\x85\xb2z^\xd5W\xa5\x99A\x94\xdf\xa4]I\xbe\xe2@\xb1\x99\xd1\xe2\xb3\x0c\x04\xa1\xfb\r\x8f/\x1b\x9e\xb4\x1a\xf8\n\xb3\x84\r\x8b*\xc1 &\xb0cph\x01\x87`\x02\xdf]\xa1\xac\x83\xcb\xe0\xbcs\x84\xc1\x12\x83Z\xa2\xbf\x99D\xf2\xae\xd5\xbf.x\x0c\x12\x1f\x9fv7Bq\x8a\x9b\x91\x0c\xdc\x0b<r1\x08l\xc4\x10\xd0}\xb8?Ys=\xb7\xde\xdc\r7c\x88h!\x99\xd2\t\x11\x9a\x014h\xdf\x84\xeb\x08+\xe0\xb5\xa2\xad\xb6\x8br\xb1(\x03K\x04\xc5\x11x-\x86\xd7i\xceX\x12+\xb0\xac\xe8\x8e \xdd\x95t6/\xc4\xf0\x06=\xf4\xff\xd7W\xe94\x1f\xf1H\xc4\xf0f\x0c\xab4\xc7\x1b\x91\x84\xb7\xf4\\\x0c\x17i\xd7\x9c\x08\xa3\xc05\x153l\xed\x1a\x83\xb7\t\xed1\x93v%%\x0f\x98\xad\xc6\xbe\x93j\xf0.J\x86\xb1\xde\xa3\x9dI\t\xb3\xdc\x13\xb0N\xb3\x13#\x10\xc7p\x89\x16Z\x12N.\xfb\xb0\xa1\xb7\xc7\xb0I{\r\x9f\x07\x08\x8bD\x10\x1a\x964\xc4\xb2\x1bFp\x19\xd5y\x9fv\xeaB\xf8\'\x90l\x08\x1f\xcc\xc6\xf0!\xed6\xa5\xe7\xd7\x10\x8b\xd1E\xb5.\x02\xf8\x08\x81\x1f\xd3nc~\xc5\x93u+\\@\xc6\xaaB\xf8\x84\xc1\x15\x9cLg\xdc\x19E>\x9f\xd2\xbe\xedi\xd4_\x91\x19\xb8h+\xcd?c\xf0\xf9\xbf\xd0/h.\x902\x82/i\xdfu\xa5\x83\xafb\xf8z\xa7<C\xf0\r\x83o\xaf#\xcfw\xa8J\xaa\xd0\xf7\x98\xe3\x07\xdaa \x15GZ\xf0#\xedhQ\x87\x9fb\xf8\x19\xef\xc7/\xb4\xd7\x12\xbe\xa8[\xa2\x1em#\r\xbf\xae\xc3o\x0c~\'\xce\x15=\x8b1\xfe\xe0\x11\xfcY\xfa\x07\x14\x1f\x95\xac'
fchapoton commented 5 years ago
comment:37

This works fine in python3:

sage: with open('State.sobj', 'rb') as f:
....:     f.read()
simon-king-jena commented 5 years ago
comment:38

Replying to @fchapoton:

This works fine in python3:

sage: with open('State.sobj', 'rb') as f:
....:     f.read()

Do I understand that it works because b means "binary"? But of course the real question is why Python 3 cannot unpickle from that file.

I tried to dig a bit deeper into Sage's load function (I guess the error is hidden there), and indeed load() uses with open(filename, 'rb') as fobj:. So, the unpickling problem is something else:

sage: from sage.matrix.matrix_gfpn_dense import mtx_unpickle
sage: filename = '/home/king/Projekte/coho/tests/State.sobj'
sage: compress = True
sage: class unpickle_old_mtx:
....:     def __call__(self, *args, **kwds):
....:         return mtx_unpickle(*args, **kwds)
....:     
sage: register_unpickle_override('pGroupCohomology.mtx', 'MTX_unpickle_class', unpickle_old_mtx)
# The following is copied from Sage's load() function.
sage:     ## Load file by absolute filename
....:     with open(filename, 'rb') as fobj:
....:         X = loads(fobj.read(), compress=compress)
....:     try:
....:         X._default_filename = os.path.abspath(filename)
....:     except AttributeError:
....:         pass
....: 
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-9-654643a51ef6> in <module>()
      1 ## Load file by absolute filename
      2 with open(filename, 'rb') as fobj:
----> 3     X = loads(fobj.read(), compress=compress)
      4 try:
      5     X._default_filename = os.path.abspath(filename)

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    967 
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 
    971 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

So, now we see that the error occurs in SageUnpickler(io.BytesIO(x)).load().

simon-king-jena commented 5 years ago
comment:39

I find it very difficult to track down. But the fact that one does see a mentioning of pGroupCohomology.mtx when one forgets to register_unpickle_override(...) seems to indicate that the error occurs when an mtx matrix is unpickled.

So, I am now trying to find if this is really the case or not.

simon-king-jena commented 5 years ago
comment:40

Replying to @simon-king-jena:

I find it very difficult to track down. But the fact that one does see a mentioning of pGroupCohomology.mtx when one forgets to register_unpickle_override(...) seems to indicate that the error occurs when an mtx matrix is unpickled.

So, I am now trying to find if this is really the case or not.

It isn't. I inserted a print statement at the beginning of mtx_unpickle, but it isn't shown before the error triggered by load('/home/king/Projekte/coho/tests/State.sobj') is raised.

I'll open a ticket for that problem.

simon-king-jena commented 5 years ago
comment:41

Replying to @simon-king-jena:

I'll open a ticket for that problem.

It is #28444. I am now convinced that the problem is in Python-3 resp. in Sage's unpickling, but not in MeatAxe.

simon-king-jena commented 5 years ago

Dependencies: #28444

simon-king-jena commented 5 years ago
comment:43

It seems that #28444 fixes the problem for p_group_cohomology on Python-3. With the current unpublished code for the package, I get

$ ./sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.9.beta9, Release Date: 2019-09-02               │
│ Using Python 3.7.3. Type "help()" for help.                        │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.set_workspace(tmp_dir())
sage: H = CohomologyRing(8,3,options='debug')
Local data found at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj
Creating symbolic links from /home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3 to /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3
Get original root and GStem
/home/king/SPKG/database 8gp3
now, StateFile is /home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3/dat/State.sobj
Try to get (8, 3) from cache
Group data are rooted at '/home/king/.sage/temp/klap/7894/dir_05fi7c3s/'
The state descriptor of the to-be-unpickled ring is expected to be provided at '/home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3/dat/State.sobj'
Setting state
> export action matrices
Differential reloaded
> rk P_02 =   3
Differential reloaded
> rk P_03 =   4
Import monomials
Checking compatibility of SmallGroups library and stored cohomology ring
sage: print(H)

Cohomology ring of Dihedral group of order 8 with coefficients in GF(2)

Computation complete
Minimal list of generators:
[c_2_2: 2-Cocycle in H^*(D8; GF(2)),
 b_1_0: 1-Cocycle in H^*(D8; GF(2)),
 b_1_1: 1-Cocycle in H^*(D8; GF(2))]
Minimal list of algebraic relations:
[b_1_0*b_1_1]

Hooray...

simon-king-jena commented 5 years ago
comment:44

Strangely, while I can compute some cohomology rings, the following computation fails in py3:

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: H = CohomologyRing(32,4)
sage: H.make()

Reason: Automatic saving does not work, and apparently it is because H.__getstate__() returns Singular elements, which cannot be pickled. It seems I did take care of it, as H.__getstate__() in py2 only returns the class sage.interfaces.singular.SingularElement and not the instance, but for some reasons it doesn't in py3.

simon-king-jena commented 5 years ago
comment:45

Question: Has the mechanism of tab completion changed between py2 and py3? I notice that tab completion on a cohomology ring works in py2, but doesn't work (no error, but no completion either) in py3.

dimpase commented 5 years ago
comment:46

tab completion is done by ipython, which should be upgraded (they've switched to Python 3 only, so we're using an old version, which in turn might lag in its Python 3 support).

I suggest not to worry about it now.

simon-king-jena commented 5 years ago
comment:47

Aha! The culprit for pickle problem is pGroupCohomology.cohomology.pickle_gap_data, which inspite of its name also takes care of singular elements. In Py2:

sage: self = CohomologyRing(32,4)
sage: self.make()
sage: G = self._decorator_cache['_parameter_restrictions',frozenset({'c_2_2'}),('radical', True)]
sage: G
[[(polynomial ring, over a field, global ordering
   // coefficients: ZZ/2
   // number of vars : 2
   //        block   1 : ordering M
   //                  : names    c_1_0 c_1_1
   //                  : weights      1     1
   //                  : weights     -1     0
   //        block   2 : ordering C,
   <repr(<sage.interfaces.singular.SingularElement at 0x7ff4a0cca690>) failed: AttributeError: 'NoneType' object has no attribute 'group'>,
   <repr(<sage.interfaces.singular.SingularElement at 0x7ff4a0cca500>) failed: AttributeError: 'NoneType' object has no attribute 'group'>)],
 2]
sage: pickle_gap_data(G)
[[(<class 'sage.interfaces.singular.SingularElement'>,
   <class 'sage.interfaces.singular.SingularElement'>,
   <class 'sage.interfaces.singular.SingularElement'>)],
 2]

In Py3:

sage: pickle_gap_data(G)
[[(polynomial ring, over a field, global ordering
   // coefficients: ZZ/2
   // number of vars : 2
   //        block   1 : ordering M
   //                  : names    c_1_0 c_1_1
   //                  : weights      1     1
   //                  : weights     -1     0
   //        block   2 : ordering C,
   <repr(<sage.interfaces.singular.SingularElement at 0x7f6475b86a20>) failed: AttributeError: 'NoneType' object has no attribute 'group'>,
   <repr(<sage.interfaces.singular.SingularElement at 0x7f6475b86dc8>) failed: AttributeError: 'NoneType' object has no attribute 'group'>)],
 2]
simon-king-jena commented 5 years ago
comment:48

Ouch. A nasty thing.

In pickle_gap_data, I test if the input is an iterable class, and if it is and if the class is a builtin, then the pickle_gap_data traverses the items of the input and creates from the individual results a modified copy of the input. In code:

    try:
        I = iter(G)
    except:
        return G
    if isinstance(G,dict):
        return dict((pickle_gap_data(k), pickle_gap_data(v)) for k,v in G.items())
    if getattr(type(G), '__module__', None) == '__builtin__':
        return type(G)(pickle_gap_data(X) for X in I)
    return G

The problem is that in Py3 we get

sage: getattr(list, '__module__', None)
'builtins'

Question: How to test in a py2/py3 invariant way whether a type is a builtin?

That asked, are there more iterable builtins than list, tuple, str/bytes and dict? I guess it is safer to just explicitly test for list and tuple. EDIT Like this:

    if isinstance(G, dict):
        return dict((pickle_gap_data(k), pickle_gap_data(v)) for k,v in G.items())
    if isinstance(G, (list, tuple)):
        return type(G)(pickle_gap_data(X) for X in G)
    return G
dimpase commented 5 years ago
comment:49

An iterable is an object with an __iter__ method or/and __getitem__ method defined. So perhaps you can test for their presense. (These methods may be broken, but OK...)