Move away from http request signing

[achamayou]

Governance currently uses http request signing to authenticate governance commands. This causes two main problems:

1. the draft keeps changing, causing churn (#3011, #1866 etc)
2. client support is (very) poor

So we would like to investigate alternatives, in particular the possibility of using https://cose-wg.github.io/cose-spec/#rfc.section.4.2, and setting the result in a CCF-specific header.

The expected benefits are:

1. stability, since COSE is published
2. shorter client code, thanks to COSE libraries, independent from the HTTP layer

We want to validate 2. by writing some sample client code.

[achamayou]

There are two main top-level possibilities to decouple signing from transport:

1. Include the signature in the message body itself, using a format like COSE Sign1
2. Include the signature as a HTTP header, for example as a JWT or a CWT

The benefit of the first approach is complete transport independence. Transports that do not support headers would still work, and programming contexts that do not allow clients to easily control header setting and formatting aren't problematic. Because the body can encoded arbitrarily, this is also an efficient choice.

The second approach inevitably requires base64 encoding, which is somewhat wasteful, and the use of the Authorization header in HTTP. JWTs are however more widespread than COSE, and library support is more widely available.

[achamayou]

In the case of governance, the signed payload is used repeatedly across transactions, as the proposal or the vote is being evaluated. For that reason, it is extracted and stored in its own table.

In approach 1. where the request body contains the payload inline, this potentially leads to duplication in the KV store: the proposal would be stored once, parsed out, and again, in the middle of a COSE Sign1 frame, for offline audit purposes.

To alleviate this problem, CCF could make use of detached content, as described in https://cose-wg.github.io/cose-spec/#rfc.section.2: the headers and signatures could be store in the evidence table, away from the proposal itself. A verifier would treat the proposal as detached content.

Library support for detached content in COSE seems limited today, from a quick survey:

• COSE-JAVA: yes
• t_cose: yes
• veraison/go-cose: yes
• COSE-csharp: no
• pycose: no
• cose-js: no

This isn't a substantial issue for CCF itself, but may be a hurdle for potential ledger auditors.

[achamayou]

Proposed format for proposals and ballots:

label = int / tstr
values = any
empty_map = bstr .size 0

Generic_Headers = (
    ? 1 => int / tstr,  ; algorithm identifier
    ? 2 => [+label],    ; criticality
    ? 3 => tstr / int,  ; content type
    ? 4 => bstr,        ; key identifier
    ? 5 => bstr,        ; IV
    ? 6 => bstr,        ; Partial IV
    ? 7 => COSE_Signature / [+COSE_Signature] ; Counter signature
)

CCF_Governance_Headers = (
  "ccf_governance_action" => tstr,    ; "proposal" / "ballot" / "withdrawal" / "ack" / "recovery_share"
  ? "ccf_governance_proposal_id" => tstr ; Proposal id in CCF, set for ballots and withdrawals, but not proposals
)

header_map = {
    Generic_Headers,
    CCF_Governance_Headers
}

Headers = (
    protected : header_map,
    unprotected : empty_map
)

COSE_Sign1 = [
    Headers,
    payload : bstr / nil,   ; JSON payload
    signature : bstr
]

[achamayou]

Python experiments: https://github.com/achamayou/CCF/blob/cose_signing_authn/tests/signing.py#L188

A source of awkwardness compared to HTTP request signing is the need to redundantly indicate what the verb/url already encode, for example in the case of a POST /gov/proposals/{proposal_id}/withdraw. This seems like an unavoidable consequence of transport-independence however.

We could wonder if it is then necessary to have separate endpoints for governance submissions. That seems obviously good on the read side (eg. GET /gov/proposals/{proposal_id}/ballots/{member_id}), and staying symmetrical and compatible seems reasonable.

Another negative point is the lack of support for embedding CDDL in OpenAPI, which means that the schema and the documentation are going to get worse for these endpoints unless we do more things manually/in a non standard way. This would also have been true for JWS/JWT because of the base64 encoding, however.

[achamayou]

List of endpoints that only accept signed requests by members:

• POST /ack: JSON input, no response payload link
• POST /proposals: JSON input, JSON response payload link
• POST /proposals/{proposal_id}/withdraw no input, JSON response payload link
• POST /proposals/{proposal_id}/ballots JSON input, JSON response payload link

Other gov endpoints additionally accept signed requests, but also allow session-authed requests.

[achamayou]

I think the way this would look is, there would be a:

class MemberCOSESign1AuthnPolicy : public AuthnPolicy

Which would:

1. Parse the COSE Sign1 body
2. Verify the signature, after looking up the member by KID
3. Expose ccf_governance_action, ?ccf_governance_proposal_id and the content (ideally a std::span into the body) in a MemberCOSESign1AuthnIdentity.

Endpoints listed above would also allow this policy, next to MemberSignatureAuthnPolicy, for the time being (with an eventual deprecation deadline). Other endpoints would drop authentication requirements.

Everything else would remain unchanged, except the public:ccf.gov.history, which would allow a variant value, or more likely would be continued in a new public:ccf.gov.cose_history. Same for public:ccf.gov.acks.

[achamayou]

In anticipation of #4213, we will add a mandatory timestamp in the protected headers.

[achamayou]

Still to be done:

• [x] conversion of existing endpoints
• [x] user-quality Python library support
• [x] documentation + scurl sample replacements