My original implementation of signed persistent logs did not properly handle Delta-supporting Persistent fields in a few edge cases. Specifically, if a single Derecho Replicated object contains both a non-signed Persistent field and a signed Persistent field that has DeltaSupport enabled, it's possible for PersistentRegistry::sign() to produce the warning "Logic error: Version X was returned by getMinimumVersionAfter(), but it did not exist in any field" and then a subsequent handle_verify_request to produce the warning "Verification of version X failed!" because it attempts to verify an empty signature.
This happens when makeVersion is called on the Replicated object when only the non-signed Persistent field was updated, which means the non-signed field creates a log entry but the Delta-supporting signed field just advances its current version without creating a log entry (since there is no Delta data). Then in PersistentRegistry::sign(), getMinimumVersionAfter() will return the new version that is in the non-signed log as cur_nonempty_version, but updateSignature won't sign any data for this version because it doesn't exist in the signed field's log. Making things even worse, getMinimumLatestVersion() can't be used to properly detect the "current version" in this situation, because the getLatestVersion() method only returns the latest version that is in a log entry, not the current version recorded in the meta-header file (which is what's updated when a Delta-supporting object advances its current version).
I fixed this with several changes:
Added a new getNextSignedVersion() method in PersistentRegistry that returns the next version that exists in some signed field, skipping all the non-signed fields
Used getNextSignedVersion() instead of getNextVersionAfter() to increment the loop in PersistentRegistry::sign()
Added a new signed_num column to the DerechoSST to record the latest version that has been signed by a replica, since this may not be equal to the latest version persisted by a replica (if that latest version exists only in non-signed fields)
Changed PersistenceManager's handle_verify_request to use the new signed_num field when validating signatures from other replicas, and to not attempt to validate a signature that doesn't (yet) exist in the local replica's log
Changed handle_verify_request to verify a signature from another replica by simply doing a byte-level equality check against the local replica's signature on the same version, instead of doing a public-key verification on the signature. This is not quite related to fixing the bug but will improve performance. It's not necessary to do a public-key verification if the local replica is using the same private key to sign the same updates and thus should have exactly the same signature on the log entry.
Separated Replicated's persist() method into two separate persist() and sign() methods, and moved the while loop in the former persist() method up to PersistenceManager's handle_sign_request method. This is necessary to allow PersistenceManager to learn which version was signed, separately from which version was persisted, and PersistenceManager was the only caller of the Replicated::persist() method so nothing else should be impacted by this API change.
My original implementation of signed persistent logs did not properly handle Delta-supporting Persistent fields in a few edge cases. Specifically, if a single Derecho Replicated object contains both a non-signed Persistent field and a signed Persistent field that has DeltaSupport enabled, it's possible for PersistentRegistry::sign() to produce the warning "Logic error: Version X was returned by getMinimumVersionAfter(), but it did not exist in any field" and then a subsequent handle_verify_request to produce the warning "Verification of version X failed!" because it attempts to verify an empty signature.
This happens when makeVersion is called on the Replicated object when only the non-signed Persistent field was updated, which means the non-signed field creates a log entry but the Delta-supporting signed field just advances its current version without creating a log entry (since there is no Delta data). Then in PersistentRegistry::sign(), getMinimumVersionAfter() will return the new version that is in the non-signed log as cur_nonempty_version, but updateSignature won't sign any data for this version because it doesn't exist in the signed field's log. Making things even worse, getMinimumLatestVersion() can't be used to properly detect the "current version" in this situation, because the getLatestVersion() method only returns the latest version that is in a log entry, not the current version recorded in the meta-header file (which is what's updated when a Delta-supporting object advances its current version).
I fixed this with several changes: