DIP-210: Content Hashing Algorithm Changes

[wilwade]

In the spec, we have two places that are content hashes:

1. Activity Content: Hash
2. Announcement contentHash eg Broadcast Announcement

Abstract

Adding support to Announcement Content Hash for:

• Blake2b
• SHA-256

And moving to a Multihash structure

Motivation

Additional hashes allow for an easier integration on creation and support for a variety of platforms and languages, although it does increase the potential burden for consumers of the data who must support multiple hashing algorithms to be able to verify content using any hash.

Specification Pull Request

Current change pull request: #245

Rationale

We have always intended to support multiple hashes. The rational for each is different:

Chosen

• Blake2b: With the support for DSNP on Frequency, Blake2b is already used in other places
• SHA-256: A hash standard in many applications

Rejected for Now

• Blake3: Is already starting to get some use and is amazingly fast, however it is very new
• sha3-256: keccak-256, which is a pre-release of sha3-256 that is used by Ethereum, support for the official final version may be worthwhile.

Chosen Path: Multihash

Support for multihash instead of the prior v1.0 version. This brings everything into alignment with #210

Backwards Compatibility

This will break validation for all implementations. It is suggested that those who can, take advantage of using multiple algorithms for some time before using only one or more hashing algorithms in the new version.

Reference Implementation and/or Tests

None

Security Considerations

This change will allow for increased security as it decreases the likelihood that all hashing algorithms in use will be broken at the same time.

Dependencies

• None.

Additional Options

Options for Announcement contentHash

1. Switch to Blake2b-256 instead of keccak
2. Find some way to support both, likely with a prefix
3. Stick with just keccak
4. Switch to something else

References

• RFC 6234
• RFC 7693

Copyright

Copyright and related rights waived via CC0.

[wilwade]

Frequency // DSNP Spec 2022-10-13 Action items:

• WW will create a separate issue around Activity Content Hash as that is easier to add and not controversial.
• We will investigate Option 2 to find a way to support multiple hashes or easier way to version it.

[wesbiggs]

You may want to consider PHC format, which seems to have evolved from crypt(3) notation.

The advantage is future-proofing for future algorithms and future use cases, including salted/parameterized algorithms. The disadvantage is verbosity (and from what I can tell, a lack of existing standardization for algorithm names).

In its simplest form: $keccak-256$$47173285a8d7341e5e972fc677286384f802f8ef42a5ec5f03bbfa254cb01fad

This forms a variable-length prefix of at least 4, and in this case, 13 characters.

[wilwade]

You may want to consider PHC format, which seems to have evolved from crypt(3) notation.

The advantage is future-proofing for future algorithms and future use cases, including salted/parameterized algorithms. The disadvantage is verbosity (and from what I can tell, a lack of existing standardization for algorithm names).

In its simplest form: $keccak-256$$47173285a8d7341e5e972fc677286384f802f8ef42a5ec5f03bbfa254cb01fad

This forms a variable-length prefix of at least 4, and in this case, 13 characters.

Interesting. We've tried really hard to keep the data in announcements as small as possible, although repeated data is fairly compressible.

[wilwade]

Frequency // DSNP Spec 2022-10-27 Notes:

• Options around content hash
  • A. Only support one
  • Requires adding additional Announcement types when we want to add or change the hashing algorithm
  • B. Make the hashing algorithm implementation specific
  • Has issues in a multi-chain environment. Would require knowledge of the original source to be able to know which algorithm to use.
  • C. Add an enum
  • separate field
  • 33 byte contentHash with the first byte being the enum for the hash type
  • D. PHC format
  • Rejected
  • Hash defined in the referenced content
• Straw polling gives most support for C/D aka the ability to specify more than one hash.
• Important to support a LIMITED number of multiple hashes
• It would be nice if it was the SAME or subset of algorithms as with Activity Content Hashes

[wesbiggs]

At the risk of unnecessarily prolonging this discussion, I'd like to consider multihash as well. This seems to have a certain amount of traction in the wider blockchain community.

keccak-256 hashes would be represented as 0x1b20{hash} (total of 34 bytes) blake2b-256 hashes would be represented as 0xa0e40220{hash} (total of 36 bytes)

Pros:

• extensible for new hashes
• well defined mapping table
• more concise than PHC
• allows for binary encoding
• more predictable (though not fixed) length
• well-maintained libraries in many languages (Rust, etc.)

An additional consideration for this (as well as options C and D) is that prefix bytes should ideally be removed before using the hash value in a Bloom filter. In the case of multihash, the encoded value would need to be parsed so the prefix length could be determined. Edit: upon reflection, I don't see a use case to put individual hash values in a bloom filter, so this isn't particularly relevant.

[wesbiggs]

Multihash is also directly compatible with (is a component of) CIDs as used in IPFS.