Use JWS and JWK for integrity check

[maggarwal13]

This pull request provides an alternative way to verify integrity using JSON Web Signature (JWS) [RFC7515] and JSON Web Key (JWK) [RFC7517] instead of using a separate signature file for update notification file.

This method allows us to avoid race condition with update notification file and update signature file as JWS sends the signature and payload together.

This pull request also suggest to use JWS for verifying snapshot file and delta files instead of using hash in update notification file. So we no longer need to send the hash in update notification file.

Brief explanation of how JWS with JWT will work:

1. The response consists of three parts : Header, Payload and Signature

2. Each part will be encoded in Base64URL separated by dot.

3. Header will contain basic information like algorithm used to generate the Keys. For example: {"alg":"ES256"}

4. Payload will have one of the NRTM file depending on the request

   • In case of Update Notification file , it will be in the JavaScript Object Notation (JSON) format [RFC8259].
   • If it is Snapshot File , payload will be in the JSON Text Sequences [RFC7464] format and GIZIPPED
   • For Delta Files, payload will be in the JSON Text Sequences [RFC7464] format

5. Third part will be signature client must verify signature using public key.

[mxsasha]

Finally had time to look at this. Thoughts:

• Integrating the signature and content of the UNF is definitely worthwhile, due to the race condition. We did consider combining these, but inventing our own format just became real sloppy. JWS offers us this in a standard method. It saves a bunch of cleanup as well from old signature files.
• Another significant benefit is backward/forward compatibility with crypto upgrades in the future.
• There is good python library support for my end, I hacked something together and it was pretty easy.
• There may be some overhead involved. I did not measure it. Ed25519 is definitely also not instant, so I'm not concerned.
• In your PR you've also replaced the existing SHA256 hashes of the Delta/Snapshot files, where it feels like SHA256 should be faster than Ed25519/JWS. And there is no race condition issue there.

I think we should:

• Adopt JWS for the signature of the UNF.
• Have update-notification-file.json be a JWS Compact Serialization, where the payload is the actual UNF content, i.e. a bytestring that decodes as JSON. See my gist for a quick python demo.
• Keep the formats of Delta/Snapshot as they are, including recording their SHA256 hash in the UNF.
• Require the use of EC keys and forbid the use of the "none" signature algorithm. Not sure if we need other tightening.
• Define the exact format in which the key should be published, both in the next_signing_key field and other publication. Just using PEM makes most sense, the keys aren't huge and not frequently transferred.

Any objections from @job @stkonst ?

For editing, please edit only the XML file - that is the source from which all others are generated.

[maggarwal13]

Hi Sasha,

Thanks for looking into it.

The reason I changed the Delta/Snapshot as well because I thought for consistency and maintenance purpose it may be good to have only one way of validating the integrity of files either with JWS or using hash.

I have not compared the performance between two though calculating the hash value for entire snapshot file and then comparing with UNF also feels little bit extra step and overhead.

If you are still open I can do a quick performance comparison between two methods as Java also provides a library for JWS.

Regards, Mahesh

On 25 Sep 2024, at 20:53, Sasha Romijn @.***> wrote:

Finally had time to look at this. Thoughts:

Integrating the signature and content of the UNF is definitely worthwhile, due to the race condition. We did consider combining these, but inventing our own format just became real sloppy. JWS offers us this in a standard method. It saves a bunch of cleanup as well from old signature files. Another significant benefit is backward/forward compatibility with crypto upgrades in the future. There is good python library support for my end, I hacked something together https://gist.github.com/mxsasha/87f387318297fd38503b9e630ec76d49 and it was pretty easy. There may be some overhead involved. I did not measure it. Ed25519 is definitely also not instant, so I'm not concerned. In your PR you've also replaced the existing SHA256 hashes of the Delta/Snapshot files, where it feels like SHA256 should be faster than Ed25519/JWS. And there is no race condition issue there. I think we should:

Adopt JWS for the signature of the UNF. Have update-notification-file.json be a JWS Compact Serialization, where the payload is the actual UNF content, i.e. a bytestring that decodes as JSON. See my gist https://gist.github.com/mxsasha/87f387318297fd38503b9e630ec76d49 for a quick python demo. Keep the formats of Delta/Snapshot as they are, including recording their SHA256 hash in the UNF. Require the use of EC keys and forbid the use of the "none" signature algorithm. Not sure if we need other tightening. Define the exact format in which the key should be published, both in the next_signing_key field and other publication. Just using PEM makes most sense, the keys aren't huge and not frequently transferred. Any objections from @job https://github.com/job @stkonst https://github.com/stkonst ?

For editing, please edit only the XML file - that is the source from which all others are generated.

— Reply to this email directly, view it on GitHub https://github.com/mxsasha/nrtmv4/pull/39#issuecomment-2374910478, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL7HNBCPLIHXNAHJWL42MATZYMBEFAVCNFSM6AAAAABLH546WKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZUHEYTANBXHA. You are receiving this because you authored the thread.

[mxsasha]

Yes, some data would be good. We should compare deltas and snapshots for RIPE db, using EC keys and compact format, for time spent for signing and for verification. File size also matters, I imagine the base64 step adds a bit of overhead. If we would want to use JWS for all files, we should also measure if it's better to gzip the snapshot payload, then JWS, or first sign, then gzip.

[mxsasha]

I've run some numbers from my end, for the server side, on a 350MB RIPE gzipped snapshot: JWS is about 5 times slower than sha256, and requires about 100 times more memory.

Some of these are limitations of the JWS libraries available: they seem to be pure Python, and they don't seem to able to process the payload in chunks. Some of these may be possible with a new implementation. However, as the two known implementations of NRTMv4 are Python and Java, we should try to make it reasonably easy to implement in both, and JWS does not meet this for large files.

My conclusion: I still agree with my notes above on keeping the SHA256 hashes for delta/snapshot files.

[maggarwal13]

I also ran some tests on my end. For snapshot I zipped first and then signed using JWS. Base64 does increase the size of the file which is almost twice. Though I did not see such a huge memory spike.

I also agree with you now, to go for JWS for UNF and keep SHA256 hashes for delta/snapshot files.

On 4 Oct 2024, at 12:19, Sasha Romijn @.***> wrote:

I've run some numbers from my end, for the server side, on a 350MB RIPE gzipped snapshot: JWS is about 5 times slower than sha256, and requires about 100 times more memory.

Some of these are limitations of the JWS libraries available: they seem to be pure Python, and they don't seem to able to process the payload in chunks. Some of these may be possible with a new implementation. However, as the two known implementations of NRTMv4 are Python and Java, we should try to make it reasonably easy to implement in both, and JWS does not meet this for large files.

My conclusion: I still agree with my notes above on keeping the SHA256 hashes for delta/snapshot files.

— Reply to this email directly, view it on GitHub https://github.com/mxsasha/nrtmv4/pull/39#issuecomment-2393363365, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL7HNBBVX5LBIGF5TJO42OTZZZTTZAVCNFSM6AAAAABLH546WKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJTGM3DGMZWGU. You are receiving this because you authored the thread.

[job]

I also agree with you now, to go for JWS for UNF and keep SHA256 hashes for delta/snapshot files.

Yeah, I too agree.

[mxsasha]

I've picked this up based on the XML file instead of txt, and addressing various finer details, in the new PR #40