test: verify integrity of message transfer from warpcast to hubs

[varunsrin]

What is the feature you would like to implement? Verify that Warpcast has correctly transferred all valid messages produced by users over a time period into Hubble with no unexplained errors.

Why is this feature important? Knowing that messages are being published correctly is a pre-requisite to opening up the Hub API's to third party developers.

Will the protocol spec need to be updated? No

How should this feature be built? (optional) The test suite should periodically compare all messages present on Hubble with all messages present in the Warpcast backend, and throw errors if Hubble has any messages that are not present in Warpcast. After @sds has uploaded all messages into Warpcast, we should verify that this runs error free for a period of 24 hours before closing out the ticket.

[varunsrin]

after live discussion, a better approach would be using a background job to pick a starting timestamp, and do a diff of all items for a user before that timestamp between hub state and warpcast backend state. doing this as a background job on warpcast makes shipping much faster at the expense of not integration testing a small portion of the api, and doing a diff of all time is easier because there is no easy way to query hubs by time ranges.

[varunsrin]

@CassOnMars is currently working on this

[varunsrin]

@CassOnMars is it correct to assume that with the way youve designed the test suite, the follow condition would produce an error that would be flagged?

1. User generates 10 messages on Warpcast, which are both under the max size limit and max age limit on Hubs
2. Warpcast broadcasts messages 1 through 9, but fails to broadcast message 10

[CassOnMars]

The current behavior is different, that instead it logs an error a message that is on the hub and not the backend, the alternative direction is logged as info – it is possible the backend would have messages a hub does not due to pruning. I could alter it such that it is an error instead.

[varunsrin]

i think it might be worth logging them as errors, but with some simple logic to eliminate most false positives: if the timestamp of the message is < 1 year ago, and the message is on backend but not hubs log error, not info

this will still throw false positives for users like @dwr who have > 10k messages and are getting pruned for max-size limits. for these cases we can either create a manual list of users with >10k messages and exclude them, or do it dynamically if we know that a particular message is greater than 10,000 messages old.