Invalid and missing signatures on webhook requests

[gareth-evans]

Describe the bug I've found that a number of events have failed validation when trying to verify the x-signature header.

I have two examples of document.created events that happened 4 minutes apart where the signature verification succeeds for one but fails for the other.

This isn't exclusively for documents, as the same behaviour has been observed on vendor and contact events as well.

Possibly unrelated but we are also seeing events coming through without x-signature header and the legacy reapit webhook header.

To Reproduce I can provide a repro in C# using the prod event payloads and signatures if required.

Expected behaviour Signature should always be present and valid.

Specification

• Project: Reapit.Services.Webhooks
• Look at the examples provided (see comments below) and try to work backwards to find out where we aren't calculating the signature correctly. I would anticipate these to both be retry events perhaps where we explicitly have to recalculate the signature due to the SendAttempts being incremented. Examples can be located in the webhook events dynamodb table
• This may relate to a previous fix where we didn't recalculate the signature for retried events. This can be seen in this PR

[github-actions[bot]]

Thank you for taking the time to report a bug. We prioritise bugs depending on the severity and implications, so please ensure that you have provided as much information as possible. If you haven’t already, it really helps us to investigate the bug you have reported if you provide ‘Steps to Replicate’ and any associated screenshots. Please ensure any personal information from the production database is obscured when submitting screenshots. This issue will be reviewed in our weekly refinement sessions and assigned to a specific project board. We may also update the ticket to request additional information, if required. For more information on our processes, please click here

[plittlewood-rpt]

HI @gareth-evans please can you send the event ids so we can look at this? That should be all we need I think

[gareth-evans]

Thanks for coming back so quickly @plittlewood-rpt

Working Event: 30643d37-8dd0-4df1-9bc4-5499fa68fc74 Not Working Event: acd153cd-887d-44e8-b9f3-dd9a2eb77e43

[gareth-evans]

There is also the following event where neither the x-signature or the legacy signature header are present

4a661460-2893-466c-896a-a198575445a0

[cduggan-reapit]

Hi @gareth-evans,

Thanks for sharing those event IDs.

Regarding the first example that failed verification, would you mind sharing your signature verification code with us to help investigate?

We've also seen a few empty header events in recent weeks. I'm not sure what the root cause is yet, but we hope to release an update shortly which will help narrow this down.

[github-actions[bot]]

We have recently requested additional information relating to the issue you have raised. Please can you take the time to review this ticket and where applicable, provide the information requested. For more information on our processes, please click here

[joekeilty-oub]

Hey, we have also experienced similar issues.

Issue 1 - Received X-Signature header that contained 2 signatures (comma delimited)

Event ID: 028d5bbb-255d-4f75-9b40-cb5894359bd7

Issue 2 - X-Signature failed cryptographic signature verification

Event IDs: 9be38873-7c14-4e15-83a1-19dd3979a6be, bb23a7a3-1e0b-477d-98cc-6bf080a99350, d3686456-8717-4fab-841e-cf9cce3a89f3, d3686456-8717-4fab-841e-cf9cce3a89f3, d3686456-8717-4fab-841e-cf9cce3a89f3, 06e8f827-9e60-4937-aca9-7e725713ed52, b29713bf-9704-44d4-a59b-8aae65dd4478, b02358ad-7f55-466f-a5fa-b6e7815c9ed2, c9b41e86-be28-4f23-89d5-8732d64b6552, aa7ef128-1d95-4cd6-b26e-3e34babd37b3, aa7ef128-1d95-4cd6-b26e-3e34babd37b3

Around 0.012% of our webhooks are failing cryptographic verification.

Happy to share our verification code privately if required

[plittlewood-rpt]

Hi @joekeilty-oub thanks for this report. Issue 1 sounds very strange! Please can you email your verification code to sinfo@reapitfoundations.zendesk.com and we can take a look at point 2?

[plittlewood-rpt]

@AshDeeming when this comes through please can you forward to me and Craig

[joekeilty-oub]

Sent that through now 👍

[cduggan-reapit]

Hi @joekeilty-oub - there's definitely something going on with those signatures! We've temporarily increased the logging from our webhook processor to help find the root cause; could you let us know if you receive any bad signatures after 14:30 BST today?

[joekeilty-oub]

Will do

[joekeilty-oub]

Didn't get a cryptographic failure but got another webhook with 2 signatures: 92ec3d4b-b93f-46a0-b7ab-898dbfdc7ab9

[joekeilty-oub]

Cryptographic failure: d4f8f018-5a53-4ad2-87b1-dbe1bae69f5a

[plittlewood-rpt]

Thanks for these Joe!

[cduggan-reapit]

Hi @joekeilty-oub, thanks for bearing with us on this. We rolled out an update that we hope will resolve your issue.

I've reviewed the logs for your webhooks since the change was affected, and it looks like they were all signed correctly. We'll continue to monitor this for the next few days, but please let us know if you receive further events that fail verification!

[joekeilty-oub]

Thanks Craig for working on the fix, I'll keep an eye out and see if any more crop up

[joekeilty-oub]

Nothing over the weekend, looks resolved :)

[cduggan-reapit]

That's great news! I'll close this ticket down - if they start cropping up again, please do let us know!

[github-actions[bot]]

It looks like you have commented on a closed issue. If your comment relates to a bug or feature request, please open a new issue, and include this issue number/url for reference. For more information on our processes, please click here