search

urda / nistbeacon

Python 3 Library to access the NIST Randomness Beacon
https://urda.github.io/nistbeacon/
Apache License 2.0
10 stars 2 forks source link

Create tests to give insight into signatureValue changes #26

Closed urda closed 7 years ago

urda commented 7 years ago

We should have some type of test or tests that can:

Why?

To determine how the library is currently behaving with these signed, but invalid records being reported by NIST. This will allow us to:

urda commented 7 years ago

It looks like the beacon.nist.gov cert has a Not Valid After of 2017-05-07T14:18:36+00:00

NIST Valid Record Discovery

Timestamp Epoch as NIST Record Link Notes
2017-05-07T14:18:00+00:00 1494166680 This could be the last valid record currently
2017-05-07T14:18:36+00:00 1494166716 This is the record at the time of Not Valid After on the certificate.
2017-05-07T14:19:00+00:00 1494166740 This is the record that is returned by the API given our Not Valid After date

NIST Valid Record Value Notes

Timestamp Output Value Previous Output Value
1494166680 29DA3AC0491B296FDCD7FB5433ECEBCF965EA10207D7A33476B1845CE05F8305F23BD38508D59F04C54A1972635A726072CC2E48DF6260951D314388FEEAB088 75C85E3EAD2C5F813C596A9EFD9AE4B4D87762C0E707C85293E433B81D20AD92CB7260BCC7E5F0AACE79DC0DBF7C2580D994D861CA5D3236D07503E3AC6E1EB6
1494166740 01BE623A11E19BA6C4FA9078A531FBE74B71AA8D64A99F1303698DF8B7712F30AB689AE92E6ADDD45D1155542DF0C9E5B2748B09286CB0A4EF19B47B726522E9 29DA3AC0491B296FDCD7FB5433ECEBCF965EA10207D7A33476B1845CE05F8305F23BD38508D59F04C54A1972635A726072CC2E48DF6260951D314388FEEAB088

As expected, record 1494166740 points to 1494166680

urda commented 7 years ago 
In [1]: from nistbeacon import NistBeacon

In [2]: record_1494166680 = NistBeacon.get_record(1494166680)

In [3]: record_1494166740 = NistBeacon.get_record(1494166740)

In [4]: record_1494166680.valid_signature
Out[4]: True

In [5]: record_1494166740.valid_signature
Out[5]: True

In [6]: record_1494166800 = NistBeacon.get_record(1494166800)

In [7]: record_1494166800.valid_signature
Out[7]: True

I expected record 1494166740 to demonstrate the current problem. I'll have to dig a bit deeper for now.

urda commented 7 years ago

Using python to find the failure boundary

#!/usr/bin/env python

from nistbeacon import NistBeacon

def seek_boundary(timestamp: int):
    max_fails = 60
    fail_count = 0
    record = NistBeacon.get_record(timestamp)

    while fail_count < max_fails:
        # Signature check
        if type(record.valid_signature) is not bool or not record.valid_signature:
            fail_count += 1

        # Report record information, and program state
        print(
            "Timestamp: {0} - "
            "valid_signature: {1} - "
            "type(valid_signature): {2} - "
            "fail_count: {3}".format(
                record.timestamp,
                record.valid_signature,
                type(record.valid_signature),
                fail_count,
            )
        )

        # Advance our timestamp pointer
        record = NistBeacon.get_next(record.timestamp)

    print("Maximum failures reached. Stopping.")

if __name__ == '__main__':
    seek_boundary(1495836840)

python program output:

$ ./find_fail.py 
Timestamp: 1495836840 - valid_signature: True - type(valid_signature): <class 'bool'> - fail_count: 0
Timestamp: 1495836900 - valid_signature: True - type(valid_signature): <class 'bool'> - fail_count: 0
Timestamp: 1495836960 - valid_signature: True - type(valid_signature): <class 'bool'> - fail_count: 0
Timestamp: 1495837080 - valid_signature: True - type(valid_signature): <class 'bool'> - fail_count: 0
Timestamp: 1496176860 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 1
Timestamp: 1496176920 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 2
Timestamp: 1496176980 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 3
Timestamp: 1496177040 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 4
Timestamp: 1496177100 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 5
Timestamp: 1496177160 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 6
Timestamp: 1496177220 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 7
Timestamp: 1496177280 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 8
Timestamp: 1496177340 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 9
Timestamp: 1496177400 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 10
Timestamp: 1496177460 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 11
Timestamp: 1496177520 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 12
Timestamp: 1496177580 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 13
Timestamp: 1496177640 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 14
Timestamp: 1496177700 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 15
Timestamp: 1496177760 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 16
Timestamp: 1496177820 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 17
Timestamp: 1496177880 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 18
Timestamp: 1496177940 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 19
Timestamp: 1496178000 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 20
Timestamp: 1496178060 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 21
Timestamp: 1496178120 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 22
Timestamp: 1496178180 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 23
Timestamp: 1496178240 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 24
Timestamp: 1496178300 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 25
Timestamp: 1496178360 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 26
Timestamp: 1496178420 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 27
Timestamp: 1496178480 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 28
Timestamp: 1496178540 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 29
Timestamp: 1496178600 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 30
Timestamp: 1496178660 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 31
Timestamp: 1496178720 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 32
Timestamp: 1496178780 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 33
Timestamp: 1496178840 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 34
Timestamp: 1496178900 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 35
Timestamp: 1496178960 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 36
Timestamp: 1496179020 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 37
Timestamp: 1496179080 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 38
Timestamp: 1496179140 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 39
Timestamp: 1496179200 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 40
Timestamp: 1496179260 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 41
Timestamp: 1496179320 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 42
Timestamp: 1496179380 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 43
Timestamp: 1496179440 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 44
Timestamp: 1496179500 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 45
Timestamp: 1496179560 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 46
Timestamp: 1496179620 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 47
Timestamp: 1496179680 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 48
Timestamp: 1496179740 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 49
Timestamp: 1496179800 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 50
Timestamp: 1496179860 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 51
Timestamp: 1496179920 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 52
Timestamp: 1496179980 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 53
Timestamp: 1496180040 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 54
Timestamp: 1496180100 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 55
Timestamp: 1496180160 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 56
Timestamp: 1496180220 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 57
Timestamp: 1496180280 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 58
Timestamp: 1496180340 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 59
Timestamp: 1496180400 - valid_signature: 0 - type(valid_signature): <class 'int'> - fail_count: 60
Maximum failures reached. Stopping.

Why 1495836840 ?

I was using the API to perform a binary search to determine where the failures started. Eventually, 1495836840 got me close enough to determine the problem area.

urda commented 7 years ago
Epoch Timestamp valid_signature value valid_signature type
1495837080 Friday, 26-May-17 22:18:00 UTC True <class 'bool'>
1496176860 Tuesday, 30-May-17 20:41:00 UTC 0 <class 'int'>

Looks like there was a window of downtime. But 1495837080 is well after the Not Valid After date. Hm.

urda commented 7 years ago

It looks like the signature_value has dropped from a length of 512 to 256.

urda commented 7 years ago

I can safely determine this is not because of the expired certificate, but NIST has changed how signatures are computed at this time.

grempe commented 7 years ago

@urda Did you ever determine what the cause of the drop of signature_value length from 512 to 256 was caused by? I'd love to be able to implement a workaround to validate signatures after 1495837080

urda commented 7 years ago

Hi @grempe !

I'm still engaged with NIST at this time regarding this change and their upcoming API changes. I've been hesitant to fully patch out record validation until then. Alas, the best advice I can offer you for now is to either wait for me to push a library update, or provide more information from NIST as it becomes available to me.

grempe commented 7 years ago

ok, I asked Rene for an update via email just now as well (you were cc). We'll see if there is a response. Thx

urda commented 7 years ago

@grempe great thanks so much! I was about to ping him again as well.

grempe commented 7 years ago

Hi @urda, FYI, I have implemented the new pubkey.pem file that Rene provided you and I in email, and I am able to verify the old and new block signatures. I've coded it in such a way as to try the new pem key first, and then fallback to trying to verify with the x509 cert if that fails. Only if they both fail will I mark the record with a bad signature.

I believe there is still a segment of blocks from during the recent HSM crash period where they cannot verify with either key. I have not yet walked the whole chain to identify that segment though. Since you already have a script to perform that chain walk perhaps you'll give it a go so we can at least document which block ranges will never verify.

You can take a look over here for my JS lib.

https://github.com/grempe/nist-randomness-beacon

Cheers.

urda commented 7 years ago

This is my current understanding as I'm working on my library's patch:

Milestone Timestamp
START ... Original Cert 1378395540
END ..... Original Cert 1495837080
START ... Invalid Signatures 1496176860
END ..... Invalid Signatures 1502201640
START ... New Cert 1502202360
END ..... New Cert All records following

Without doing any research, I imagine all records that fall between the START and END of "Invalid Signatures" we will be unable to match due to NIST's bug in record signing.

What to do with this info?

Means a few things:

urda commented 7 years ago

Fixed and released.