Page type mismatch, expected 4/OID_INDEX (tx=0) but got 0, PageId=718102

[tzaeschke]

Encountering an (unexpected) page type 0 happens when ZooDB tries to read beyond the end of the database file, i.e. it reads a 0 from an empty page.

This happened in issue #109 in a user's software. The fact that the database file generated by the software is exactly 2GB in size (smaller than the requested page ID indicates) suggests that the database file is incomplete. The current working theory is that this is due to the software making an incomplete copy of the database file. PageId 718102 would require at least a 2.9GB database file (4KB per page).

I suggested a fix form compomics utilities here.

Previous text (outdated)

ZooDB uses Java's MappedByteBuffer for reading files. Unfortunately, MappedByteBuffers cannot access files > 2GB (see for example the position(int) method).

~~This should be easily fixable. If it requires a format change, we should first implement a mechanism that prevents files > 2GB instead of failing silently.~~

Correction: ZooDB doesn't use MappedByteBuffer (not anymore). Still, something goes wrong when writing > 2GB. In fact, there are example of working database files > 5GB. But one user's files stop working suspiciously close to 2GB.

[hbarsnes]

Thanks for figuring out the source of the problem!

I've already updated our code, but I cannot yet test the fix above as I now get a different issue when trying to reopen saved PeptideShaker projects. The below error occurs both with ZooDB 0.5.2 and 0.6.0.

Basically, we have the following HashMap

HashMap<String, HashSet<Long>> proteinMap = new HashMap<>();

which can be added to our ZooDB database without any issues (of course with content added first), but when trying to extract the map from the database (after saving the PeptideShaker project) I get the following error:

java.lang.RuntimeException: Field: class java.util.HashMap proteinMap
    at org.zoodb.internal.DataDeSerializer.deserializeFields2(DataDeSerializer.java:417)
    at org.zoodb.internal.DataDeSerializer.readObjPrivate(DataDeSerializer.java:304)
    at org.zoodb.internal.DataDeSerializer.readObject(DataDeSerializer.java:182)
    at org.zoodb.internal.server.index.ObjectPosIterator.findNext(ObjectPosIterator.java:64)
    at org.zoodb.internal.server.index.ObjectPosIterator.<init>(ObjectPosIterator.java:46)
    at org.zoodb.internal.server.DiskAccessOneFile.readAllObjects(DiskAccessOneFile.java:284)
    at org.zoodb.internal.model1p.Node1P.loadAllInstances(Node1P.java:136)
    at org.zoodb.internal.Session.loadAllInstances(Session.java:472)
    at org.zoodb.internal.Session.loadAllInstances(Session.java:477)
    at org.zoodb.internal.Session.loadAllInstances(Session.java:477)
    at org.zoodb.internal.Session.loadAllInstances(Session.java:455)
    at org.zoodb.internal.query.ClassExtent.iterator(ClassExtent.java:113)
    at org.zoodb.internal.query.QueryExecutor.applyQueryOnExtentV3(QueryExecutor.java:490)
    at org.zoodb.internal.query.QueryExecutor.runQueryV3(QueryExecutor.java:148)
    at org.zoodb.internal.query.QueryExecutor.runQuery(QueryExecutor.java:122)
    at org.zoodb.jdo.impl.QueryImpl.runQuery(QueryImpl.java:549)
    at org.zoodb.jdo.impl.QueryImpl.execute(QueryImpl.java:535)
    at com.compomics.util.db.object.ObjectsDB.establishConnection(ObjectsDB.java:916)
    at com.compomics.util.db.object.ObjectsDB.establishConnection(ObjectsDB.java:883)
    at com.compomics.util.db.object.ObjectsDB.<init>(ObjectsDB.java:131)
    at eu.isas.peptideshaker.utils.PsdbParent.loadPsdbFile(PsdbParent.java:220)
    at eu.isas.peptideshaker.gui.PeptideShakerGUI$104.run(PeptideShakerGUI.java:6563)
Caused by: java.lang.IllegalArgumentException: Can not set final java.util.HashMap field com.compomics.util.experiment.identification.IdentificationKeys.proteinMap to java.lang.Long
    at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
    at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
    at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.set(UnsafeQualifiedObjectFieldAccessorImpl.java:83)
    at java.lang.reflect.Field.set(Field.java:764)
    at org.zoodb.internal.DataDeSerializer.deserializeFields2(DataDeSerializer.java:412)
    ... 21 more

Do you have any idea what would cause such an issue? I have confirmed that the content of the map (upon database insertion) is correct.

If it can help to look at the PeptideShaker and utilities code please see the branches called new_backend, i.e. https://github.com/compomics/peptide-shaker/tree/new_backend and https://github.com/compomics/compomics-utilities/tree/new_backend.

I can of course also send you the data required to reproduce the error if that would help.

Thanks in advance!

[tzaeschke]

Hi, sorry, I only saw the issue today. I could import, save and load a large database fine with the version of new_backend that I checked out in February. May have something changed since then that affect this? I will try to have a look tomorrow and retry with the latest version.

[hbarsnes]

Thanks! Yes, it is most likely something that changed in our code, as I can no longer even reopen small projects. But I simply cannot figure out what change would result in the above IllegalArgumentException? Especially given that the object added to the database seems to be fine and the error only occurs when I later try to extract it?

[tzaeschke]

I checked out the latest version on new_branch, but I cannot reproduce the problem, instead the program just hangs at 8% when importing my example. Maybe you could send me a more recent example file(s)?

Having said that, it looks like a problem in ZooDB, I will have a look, maybe it is easy to find.

[hbarsnes]

Maybe you could send me a more recent example file(s)?

Of course, here's the SearchGUI output I've used in my recent testing: http://vedlegg.uib.no/?id=c3dfb40f94fe14ca2fa0b3cc49c8cff2

[tzaeschke]

I created a new issue for this: #128

Also, I added a question to the new Issue.

[hbarsnes]

I can confirm that (if loading with one thread, as suggested in https://github.com/tzaeschke/zoodb/issues/128) the "Page type mismatch" issue is resolved when reopening large saved PeptideShaker projets.

In other words, the issue was indeed due to the error in copying files larger 2GB. You may therefore close this issue. Thanks again for the help!