search

danwent / Perspectives-Server

network notary implementation for the Perspectives project
http://perspectives-project.org
GNU General Public License v3.0
50 stars 13 forks source link

Do not update previous key's end time when adding observation for new key #23

Closed daveschaefer closed 10 years ago

daveschaefer commented 11 years ago

Currently if a notary scans a site and finds a different certificate than it saw last time, it does the following:

  1. Create a new observation record for that site/certificate combination, with a start and end time of the current time
  2. Update the previous certificate observation for that site, setting the end time to the current time minus one second

We should not perform the second step - it invents data that we can't vouch for (e.g. the cert could have changed 1 second after we last observed it). It also opens the opportunity for a bad bug: if you don't run your scans regularly, or run scans after a very long absence, the notary would backfill a large time range, claiming the certificate had been seen throughout that time period.

daveschaefer commented 10 years ago

After some discussion on the list: if we don't update the previous cert at all we get cases where sites that change their certificates frequently (e.g., every observation) end up with a number of observations with one-second lengths. To avoid this but still prevent data corruption we will start by putting a limit on how far data can be backfilled.

If the previous observation ended within 48 hours, it will be updated. Otherwise it is left untouched and a new observation record is created. This allows for normal behaviour in the cases where site scans are not performed exactly every 24 hours (e.g. the scan starts at the same time but sites are scanned in a random order).

daveschaefer commented 10 years ago

Fixed in 3.2

daveschaefer commented 8 years ago

Note that this bug combined with #25 has left the data for the default notaries in a bad state. The default notaries are running a very old version of the server software (v2.0). Because their logs would fill the entire hard disk, they would be out of commission for long periods of time, and then backfill large swaths of observation data when they came back online.

I fortunate to be able to chat with both @danwent and David Andersen, who both agreed that if we can get the default notaries back online it would be best to simply start over from scratch with fresh data. I can upload a copy of the current v2 notary data for historical purposed in case anyone wants to take a look.