Java NullPointerException when trying to deserialize OTS file

[orbmis]

When I try to verify a timestamp using the Java library, I am running into a Java NullPointerException.

This is my code:

byte[] originalFile = Files.readAllBytes(fileData.get("absoluteFilePath"));
byte[] otsFile = Files.readAllBytes(fileData.get("otsFilePath"));

// below is from the examples...

DetachedTimestampFile detached = DetachedTimestampFile.from( new OpSHA256(), originalFile);
DetachedTimestampFile detachedOts = DetachedTimestampFile.deserialize(otsFile);

HashMap<Chains, VerifyResult> result = OpenTimestamps.verify(detachedOts,detached);

And this the exception:

java.lang.NullPointerException: null
    at com.eternitywall.ots.Timestamp.doTagOrAttestation(Timestamp.java:101) ~[java-opentimestamps-1.16.jar:na]
    at com.eternitywall.ots.Timestamp.deserialize(Timestamp.java:89) ~[java-opentimestamps-1.16.jar:na]
    at com.eternitywall.ots.DetachedTimestampFile.deserialize(DetachedTimestampFile.java:107) ~[java-opentimestamps-1.16.jar:na]
    at com.eternitywall.ots.DetachedTimestampFile.deserialize(DetachedTimestampFile.java:120) ~[java-opentimestamps-1.16.jar:na]
    at com.fmr.ots.FileTimestampController.verifyDocument(FileTimestampController.java:110) ~[classes/:na]

Are there examples for how to read in the OTS and original file, or a complete example of how to use the Java client in code?

[lvaccaro]

Hi, which version are you using? can you attach your ots file?

[orbmis]

I'm using version 1.16.

The example I posted above is a bit convoluted - this is the code I'm using now:

byte[] otsFileContent = Files.readAllBytes(fileData.get("otsFilePath"));

File initialFile = new File(fileData.get("absoluteFilePath").toString());
InputStream targetStream = new FileInputStream(initialFile);

DetachedTimestampFile detached = DetachedTimestampFile.from(targetStream);
DetachedTimestampFile detachedOts = DetachedTimestampFile.deserialize(otsFileContent);

HashMap<Chains, VerifyResult> result = OpenTimestamps.verify(detachedOts, detached);

And this is the error I'm getting now:

java.lang.NullPointerException: null
    at com.eternitywall.ots.StreamDeserializationContext.readVaruint(StreamDeserializationContext.java:55) ~[java-opentimestamps-1.16.jar:na]
    at com.eternitywall.ots.StreamDeserializationContext.readVarbytes(StreamDeserializationContext.java:76) ~[java-opentimestamps-1.16.jar:na]
    at com.eternitywall.ots.StreamDeserializationContext.readVarbytes(StreamDeserializationContext.java:72) ~[java-opentimestamps-1.16.jar:na]
    at com.eternitywall.ots.attestation.TimeAttestation.deserialize(TimeAttestation.java:47) ~[java-opentimestamps-1.16.jar:na]
    at com.eternitywall.ots.Timestamp.doTagOrAttestation(Timestamp.java:96) ~[java-opentimestamps-1.16.jar:na]
    at com.eternitywall.ots.Timestamp.deserialize(Timestamp.java:89) ~[java-opentimestamps-1.16.jar:na]
    at com.eternitywall.ots.DetachedTimestampFile.deserialize(DetachedTimestampFile.java:107) ~[java-opentimestamps-1.16.jar:na]
    at com.eternitywall.ots.DetachedTimestampFile.deserialize(DetachedTimestampFile.java:120) ~[java-opentimestamps-1.16.jar:na]

Unfortunately Github won't let me upload the ots file here, so here is a link to download it:

https://www.dropbox.com/s/4b280a66ljhxbqt/timestamptest.txt.ots?dl=0

This is the original file:

timestamptest.txt

[lvaccaro]

Hi @simbro , the ots file contains only the header (with sha256 of the original file), but no Merkle path; also the info command fails because the ots is invalid. There was a mistake during stamp procedure, as no internet connection to calendar server or wrong saving method.

[orbmis]

Thanks Luca, much appreciated!

Yes I did indeed have an error with the stamping function, I've resolved that now and it appears to be functioning, though I'm till not able to upgrade or verify.

These are the complete logs of stamping, and then upgrading a file:

Stamping:

Submitting to remote calendar https://alice.btc.calendar.opentimestamps.org
Submitting to remote calendar https://bob.btc.calendar.opentimestamps.org
Submitting to remote calendar https://finney.calendar.eternitywall.com
2018-11-18 23:09:13.567  INFO 13800 --- [nio-8080-exec-4] com.example.ots.FileTimestampController      : File sha256 hash: 064b96390bff5e3764b0957cbbe1fdaa4ac596b47ffab0161a664e12c01759de
Timestamp:
append b68f4e4251c8af843f2b76f68819d529
sha256
 -> append 66127344d79b5e419fa95054d90182f7
    sha256
    prepend 5bf1f118
    append 09247e79a62e3fb5
    verify PendingAttestation('https://alice.btc.calendar.opentimestamps.org')
 -> append 9953f201e8f4298638da588c4ea691ab
    sha256
    prepend 5bf1f119
    append 811d0108897869f8
    verify PendingAttestation('https://bob.btc.calendar.opentimestamps.org')
 -> append e1d19dfff7eacb3d74dc2080674ef0fe
    sha256
    append caf30d75013f13d86e8932e082295fe2
    sha256
    prepend 5bf1f119
    append 4cd34e8a6b1c0f43
    verify PendingAttestation('https://finney.calendar.eternitywall.com')

Upgrading:

https://bob.btc.calendar.opentimestamps.org/timestamp/5bf1f1192a65812e1bbf8c9056266182a8f2fa3e855ae5f121699da2f91810e59c7b7d76811d0108897869f8 exception java.io.FileNotFoundException: https://bob.btc.calendar.opentimestamps.org/timestamp/5bf1f1192a65812e1bbf8c9056266182a8f2fa3e855ae5f121699da2f91810e59c7b7d76811d0108897869f8
Calendar https://bob.btc.calendar.opentimestamps.org: null
null
https://finney.calendar.eternitywall.com/timestamp/5bf1f119013a731c524b830dcc6da069a34c288782adebebfbedd66930df3d3712c6f2354cd34e8a6b1c0f43 exception java.io.FileNotFoundException: https://finney.calendar.eternitywall.com/timestamp/5bf1f119013a731c524b830dcc6da069a34c288782adebebfbedd66930df3d3712c6f2354cd34e8a6b1c0f43
Calendar https://finney.calendar.eternitywall.com: null
null
https://alice.btc.calendar.opentimestamps.org/timestamp/5bf1f11885a0d1d2ccb58753b9466d28d564e3bd3d489685c55250840a93a7199a58a20709247e79a62e3fb5 exception java.io.FileNotFoundException: https://alice.btc.calendar.opentimestamps.org/timestamp/5bf1f11885a0d1d2ccb58753b9466d28d564e3bd3d489685c55250840a93a7199a58a20709247e79a62e3fb5
Calendar https://alice.btc.calendar.opentimestamps.org: null
null
Timestamp not upgraded

These are the three controller methods that handle timestamping, upgrading and verifying - I'm not sure where I'm going wrong:

@RequestMapping("/timestamp/{fileId}")
public String timestampDocument(@PathVariable("fileId") Long fileId) throws Exception {
    HashMap<String, Path> fileData = getFileDataFromId(fileId);

    byte[] data = Files.readAllBytes(fileData.get("absoluteFilePath"));

    try {
        DetachedTimestampFile detached = DetachedTimestampFile.from( new OpSHA256(), data );

        String infoResult = OpenTimestamps.info(detached);

        logger.info(infoResult);

        Files.write(fileData.get("otsFilePath"), detached.serialize());

        return infoResult;
    } catch (Exception e) {
        e.printStackTrace();

        throw e;
    }
}

@RequestMapping("/verify/{fileId}")
public HashMap<Chains, VerifyResult> verifyDocument(@PathVariable("fileId") Long fileId) throws Exception {
    HashMap<String, Path> fileData = getFileDataFromId(fileId);

    byte[] data = Files.readAllBytes(fileData.get("absoluteFilePath"));
    byte[] otsFileContent = Files.readAllBytes(fileData.get("otsFilePath"));

    DetachedTimestampFile detached = DetachedTimestampFile.from( new OpSHA256(), data );
    DetachedTimestampFile detachedOts = DetachedTimestampFile.deserialize(otsFileContent);

    HashMap<Chains, VerifyResult> result = OpenTimestamps.verify(detachedOts, detached);

    if (result == null || result.isEmpty()) {
        System.out.println("Pending or Bad attestation");
    } else {
        result.forEach((k, v) -> System.out.println("Success! " + k + " attests data existed as of "+ v));
    }

    return result;
}

@RequestMapping("/upgrade/{fileId}")
public String upgradeDocument(@PathVariable("fileId") Long fileId) throws Exception {
    HashMap<String, Path> fileData = getFileDataFromId(fileId);

    byte[] otsFileContent = Files.readAllBytes(fileData.get("otsFilePath"));

    DetachedTimestampFile detachedOts = DetachedTimestampFile.deserialize(otsFileContent);

    boolean changed = OpenTimestamps.upgrade(detachedOts);

    if(!changed) {
       System.out.println("Timestamp not upgraded");

       return "Timestamp not upgraded";
    } else {
       System.out.println("Timestamp upgraded");

       return "Timestamp upgraded";
    }
}

[lvaccaro]

Now you are able to stamp and check PendingAttestation in receipt; upgrade and verify look good, but you need to wait before upgrading.

Remember: It takes a few hours for the timestamp to get confirmed by the Bitcoin blockchain; we're not doing one transaction per timestamp.

[orbmis]

So I was able to upgrade successfully, and the logs show:

Got 1 attestation(s) from https://bob.btc.calendar.opentimestamps.org
Got 1 attestation(s) from https://finney.calendar.eternitywall.com
Got 1 attestation(s) from https://alice.btc.calendar.opentimestamps.org
Timestamp upgraded

However, even after 10 hours, I'm still seeing the "Pending or Bad attestation" in the logs. When I use the online verifier on opentimestamps.org, I get the following message:

"Bitcoin block 550636 attests existence as of 2018-11-18 GMT"

See timestamp info here

There must be something I'm doing wrong in the "verifyDocument" method in the above example, though it follows very closely the online example in the documentation.

[orbmis]

Sorry to ask more questions - but I'm just trying to understand how the verification part of this works.

When debugging the "verify" method, it seems as though it just looks at the OTS file for the attestations contained wherein. does this mean I should receive a new version of the file after upgrading? The upgrade method simply returns a boolean value for the "changed" property.

Should I just use the JavaScript library?

[orbmis]

I was finally able to solve the issue - the problem was very simple - the timestamp file that is passed into the upgrade method mutates the "timestamp" property. I needed to re-serialize and save the OTS file again, overwriting the original OTS file - then, use this new OTS file (with the attestations in it) to verify.