Issue with old(er) binary files

[simpcomp-team]

Hi,

for gap package simpcomp we use the io package to save simplicial complexes in binary format. Saving them and then reading them in is not a problem (so I assume our code works, at least in some instances).

But some binary files we produced with an older version of io (and which we still use in our complex library) now produce the error Found a self-reference to an unknown object!. As soon as this happens something breaks severely and reading other files does not produce an error anymore but loaded data is corrupted.

Here is what happens in detail:

gap> SCLibLoad(SCLib,548);
Found a self-reference to an unknown object!
#I  SCIntFunc.GeneralUnpickler: Error while unpicking attribute value of 'ComputedSCSkelExs'
Found a self-reference to an unknown object!
#I  SCIntFunc.GeneralUnpickler: Error while unpicking attribute name
Found a self-reference to an unknown object!
#I  SCIntFunc.GeneralUnpickler: Error while unpicking attribute value of '<object>'
#I  SCIntFunc.GeneralUnpickler: Error while unpicking attribute name
Error, no method found! For debugging hints type ?Recovery from NoMethodFound
Error, no 1st choice method found for `InputTextString' on 1 arguments at /home/jspreer/GAP/gap-repository/lib/methsel2.g:250 called from
InputTextString( a ) at /home/jspreer/GAP/gap-repository/lib/string.gi:652 called from
EvalString( name ) at /home/jspreer/GAP/gap-repository/pkg/simpcomp/lib/io.gi:1734 called from
SCIntFunc.GeneralUnpickler( f, c ); at /home/jspreer/GAP/gap-repository/pkg/simpcomp/lib/io.gi:1757 called from
up( f ) at /home/jspreer/GAP/gap-repository/pkg/io-4.5.4/gap/pickle.gi:275 called from
IO_Unpickle( fh ) at /home/jspreer/GAP/gap-repository/pkg/simpcomp/lib/io.gi:785 called from
...  at *stdin*:26
type 'quit;' to quit to outer loop

Our methods to communicate with io are implemented in io.gi. See, for instance, functions SCIntFunc.GeneralPickler or SCIntFunc.GeneralUnpickler.

Setup is the following:

 ┌───────┐   GAP 4.10dev-1285-g44f1a0c of today
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: x86_64-pc-linux-gnu-default64-kv5
 Configuration:  gmp 6.1.0, GASMAN, readline

Happy to provide you with more details.

[fingolfin]

In order to debug this, we need a file exhibiting the issue.

[fingolfin]

I assume I can install simpcomp and then extract the example from that.

Any idea which version of io / gap still worked for you ?

[simpcomp-team]

Hi,

one of the files causing problems is this one: https://github.com/simpcomp-team/simpcomp/blob/master/complexes/transitive/5_manifolds_36.scb, which is the file which is supposed to be loaded by SCLibLoad(SCLib,548).

I don't know when exactly the error first occurred, but I think io version 4.4.6 was still ok.

[fingolfin]

I just tried this with GAP 4.8, using the package versions bundled with it (io 4.4.4 and simpcomp 2.1.6) and got the exact same error. Likewise in GAP 4.7.9, with IO 4.4.4 and simpcomp 2.1.1

[fingolfin]

... and also in GAP 4.6.5 with IO 4.2 and simpcomp 1.6.1

[fingolfin]

It does, however, work in GAP 4.5.7 with IO 4.2 and simpcomp 1.5.4

Comparing SHA56 checksums yields this:

b6aa14b4731950b73fb30382f5e117578283f8b8696e2f99460d36e0dcedd414  ./gap.github/pkg/simpcomp/complexes/transitive/5_manifolds_36.scb
b6aa14b4731950b73fb30382f5e117578283f8b8696e2f99460d36e0dcedd414  ./gap.stable-4.10/pkg/simpcomp/complexes/transitive/5_manifolds_36.scb
b6aa14b4731950b73fb30382f5e117578283f8b8696e2f99460d36e0dcedd414  ./gap-4.9.3/pkg/simpcomp/complexes/transitive/5_manifolds_36.scb
b6aa14b4731950b73fb30382f5e117578283f8b8696e2f99460d36e0dcedd414  ./gap4r8/pkg/simpcomp/complexes/transitive/5_manifolds_36.scb
b6aa14b4731950b73fb30382f5e117578283f8b8696e2f99460d36e0dcedd414  ./gap4r7/pkg/simpcomp/complexes/transitive/5_manifolds_36.scb
b6aa14b4731950b73fb30382f5e117578283f8b8696e2f99460d36e0dcedd414  ./gap4r6/pkg/simpcomp/complexes/transitive/5_manifolds_36.scb
0e1b65070675499894acc3aa2d27de75e7f021162249b526d1057bc5d815eb38  ./gap4r5/pkg/simpcomp/complexes/transitive/5_manifolds_36.scb

In conclusion, it seems that sometimes between simpcomp 1.5.4 and 1.6.1, this data file (and perhaps others) changed, and possibly got corrupted..

[fingolfin]

As to the subsequent errors you get after this failed unpickling attempt: I suspect that you can call IO_ClearPickleCache() to fix that.

Now, one might wonder: Why doesn't IO just automatically clear the pickle cache when a new pickling or unpickling request by the user comes in?

The reason is that the design of IO's pickling and unpickling code unfortunately is so that this is virtually impossible, because the same functions are called by the "user" (resp. code using pickling), and also pickling methods. So it's difficult (impossible?) to recognize whether a call to IO_Pickle resp. IO_Unpickle was made by the user (and hence the pickling cache should be reset beforehand), or recursively by an (un)pickling function.

IMHO that's a rather serious restriction of the IO pickle interface, which should have provided different functions for the "user" resp. client code to call on the one hand, and for (un)picklers to call on the other. Alas, that ship has sailed now.

[simpcomp-team]

Ok, thanks for checking.