crypto/ecdsa: make PublicKey implement encoding.TextMarshaler/TextUnmarshaler using PEM

[kudu-rex]

What version of Go are you using (go version)?

$ go version
go version go1.12.7 darwin/amd64

and whatever play.golang.org uses.

Does this issue reproduce with the latest release?

It is the latest release as of now. It can also be reproduced here: https://play.golang.org/p/cbsOhB8lHxe

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/path/to/my/gocode"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.12.7/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.12.7/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/td/yb98xptn12d60pfrjw87kkm80000gn/T/go-build432091977=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

https://play.golang.org/p/cbsOhB8lHxe

package main

import (
    "crypto/ecdsa"
    "crypto/elliptic"
    "crypto/rand"
    "encoding/json"
    "log"
)

func main() {

    privKey, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
    jsonKey, err := json.Marshal(privKey)
    if err != nil {
        log.Fatalf("error marshalling private key: %v", err)
    }

    var retrieveKey ecdsa.PrivateKey

    if err := json.Unmarshal(jsonKey, &retrieveKey); err != nil {
        log.Fatalf("error parsing json: %s", err)
    }
}

What did you expect to see?

A key struct created from json that has just been created from the very same data structure.

The same process does succeed for RSA: https://play.golang.org/p/FFYREV4NMfv , although not on the playground, where I get "Program exited: process took too long." I did test it locally and I'm actually using it as a workaround.

What did you see instead?

error parsing json: json: cannot unmarshal object into Go struct field PrivateKey.Curve of type elliptic.Curve

I have also written the jsonKey to file and verified that it is valid json.

[odeke-em]

Thank you for this report @nulldowntime and welcome to the Go project!

So the issue here is that for ecdsa.PrivateKey we don't have custom JSON marshaler and unmarhsler methods yet the embedded field Curve is an interface as per:

https://golang.org/pkg/crypto/ecdsa/#PrivateKey

which is an embedded field https://golang.org/pkg/crypto/ecdsa/#PublicKey

and finally https://golang.org/pkg/crypto/elliptic/#Curve

of which a non-nil interface is backed by a concrete type so the JSON.marshal works alright, but not the other way unless we define some custom json.Unmarshaler https://golang.org/pkg/encoding/json/#Unmarshaler methods

You can unmarshal though to the CurveParams as per https://play.golang.org/p/NfTlvFC28Zw or inlined below

package main

import (
    "crypto/ecdsa"
    "crypto/elliptic"
    "crypto/rand"
    "encoding/json"
    "fmt"
    "log"
)

func main() {

    privKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
    if err != nil {
        log.Fatalf("Failed to generate privateKey: %v", err)
    }
    jsonKey, err := json.Marshal(privKey)
    if err != nil {
        log.Fatalf("error marshalling private key: %v", err)
    }
    type retrieve struct {
        CurveParams *elliptic.CurveParams `json:"Curve"`
    }

    rt := new(retrieve)
    if err := json.Unmarshal(jsonKey, rt); err != nil {
        log.Fatalf("error parsing json: %s", err)
    }
    fmt.Printf("Retrieved.CurveParams: %#v\n", rt.CurveParams)
}

which prints out successfully

Retrieved.CurveParams: &elliptic.CurveParams{
 P:115792089210356248762697446949407573530086143415290314195533631308867097853951,
 N:115792089210356248762697446949407573529996955224135760342422259061068512044369, 
 B:41058363725152142129326129780047268409114441015993725554835256314039467401291, 
 Gx:48439561293906451759052585252797914202762949526041747995844080717082404635286, 
 Gy:36134250956749795798585127919587881956611106672985015071877198253568414405109,
 BitSize:256,
 Name:"P-256"
}

and then for a sample of json.Unmarshaler https://play.golang.org/p/_JvAmJOBe5A

I don't think these structs are normally JSON serialized and unserialized as it has been almost a decade without json interfaces being implemented but I shall defer to the crypto experts @FiloSottile @agl and I shall retitle this issue as a feature request for json.Unmarshaler methods

[kudu-rex]

Thanks for the very detailed, and frankly awesome, answer. I'm sure being able to unmarshal ecdsa keys, in addition to rsa, would not be a bad thing :), so I naturally welcome the proposal.

My use case is somewhat arbitrary (storing ACME/Let's Encrypt account details in json format), but I can't imagine this not being useful to a lot of people as ecdsa becomes more widespread.

[FiloSottile]

I think implementing PublicKey.MarshalJSON and PublicKey.UnmarshalJSON so that they just use the name of the curve (for the common ones) would be nice. No need to actually encode the whole parameters.

[FiloSottile]

Ah, also note that unmarshaling into CurveParams will force you into the generic implementation of the curve, which will be extremely slow. It's one of the issues of the Curve/CurveParams design, and why I wish we didn't have CurveParams, really.

[FiloSottile]

I just heard of another user being confused by this. I think we should do this, by calling x509.MarshalPKIXPublicKey from PublicKey.MarshalJSON and x509.ParsePKIXPublicKey from PublicKey.UnmarshalJSON, and simply encoding the []byte like encoding/json always does, as base64.

[rsc]

JSON should not be privileged. It may make more sense to implement MarshalBinary and MarshalText.

[FiloSottile]

Yeah, even better. encoding/json unfortunately doesn't know about MarshalBinary, which feels like the most appropriate of the two, so let's do MarshalText instead? (Or do we want to open a proposal for encoding/json to use BinaryMarshaler?)

I think we should use PEM for MarshalText at this point. The newlines are kind of ugly and the malleability is unfortunate but there is no obvious justification for picking plain base64 in MarshalText.

[rsc]

OK, so it sounds like the suggestion is to implement MarshalText and use PEM as the text format.

Based on the discussion above, this seems like a likely accept.

Leaving open for a week for final comments.

[rsc]

No change in consensus, so accepted with title change.

[FiloSottile]

If this works out we might consider extending it to other PublicKey and maybe PrivateKey types, but I'm afraid we'll run into import loops with crypto/x509, where the encoding functions are.

[gopherbot]

Change https://golang.org/cl/225638 mentions this issue: ecdsa: add marshaling support PrivateKey and PublicKey types

[d1str0]

Would love to see this implemented, especially since Lego, the most popular Let's Encrypt library for Go, uses ecdsa keys by default for ACME registration. These keys should be persisted and JSON can help this process for certain storage engines.

[odeke-em]

Moving this to Go1.16 as per @katiehockman’s reply to me in https://go-review.googlesource.com/c/go/+/225638/4#message-256466dabb75abcbfd8db026060462d28978df14

[katiehockman]

This is unfortunate. I'm not seeing a way to do this with the current APIs of crypto/x509 and crypto/ecdsa. To take marshaling as an example, in order to implement crypto/ecdsa.PublicKey.MarshalText, crypto/ecdsa would need to import crypto/x509.MarshalPKIXPublicKey, which would lead to a circular dependency. There are several layers of interwoven dependencies between the two packages, which make this more complicated. The only way I can see this working is if we could export something like crypto/ecdsa.MarshalPublicKey which can then be imported by crypto/x509. But even with that implementation (which would require a proposal for a new exported function), we would still need to copy a decent amount of code from crypto/x509 to encode it correctly.

I'm definitely open to suggestions from anyone else with familiarity of the codebase, but based on my investigation, we would need to copy ~200-250 lines of code from crypto/x509 into crypto/ecdsa with how the APIs are built today. I don't think the technical debt associated with that is worth the benefit of adding a TextMarshaler/TextUnmarshaler for ecdsa public and private keys.

[odeke-em]

@katiehockman perhaps we could use an internal package of sorts?

[katiehockman]

@odeke-em Can you explain the rationale for making an internal package for this? I'm not sure how that would help the reporter in this case.

[RmbRT]

I think the idea was to extract the functionality that would lead to cyclical imports into an internal package which is then shared by both packages.

[smiletrl]

For public key, it looks like we only need to save the point (X, Y), and the Curve type. If we know what curve is for the original private key, then we only need to save the public point into persistent storage.

To verify this idea,

    privKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
    if err != nil {
        panic(err)
    }
    pub := &privKey.PublicKey

    newPub := ecdsa.PublicKey{
        Curve: elliptic.P256(),
        X:     pub.X,
        Y:     pub.Y,
    }

    log.Printf("is equal: %v", newPub.Equal(pub))

    newPub2 := ecdsa.PublicKey{
        Curve: elliptic.P384(),
        X:     pub.X,
        Y:     pub.Y,
    }

    log.Printf("is equal 2: %v", newPub2.Equal(pub))

here's the result

2024/07/25 15:39:10 is equal: true
2024/07/25 15:39:10 is equal 2: false

The result shows we only need to save the point for public key, when we know the Curve. So here comes one solution

import (
    "crypto/ecdsa"
    "crypto/elliptic"
    "crypto/rand"
    "encoding/json"
    "log"
    "math/big"
)

type P256PublicKey struct {
    X, Y *big.Int
}

func (p *P256PublicKey) Key() *ecdsa.PublicKey {
    return &ecdsa.PublicKey{
        Curve: elliptic.P256(),
        X:     p.X,
        Y:     p.Y,
    }
}

func main() {
    privKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
    if err != nil {
        panic(err)
    }
    pub := &privKey.PublicKey

    key := P256PublicKey{
        X: pub.X,
        Y: pub.Y,
    }

    // Get bytes and save
    bz, err := json.Marshal(key)
    if err != nil {
        panic(err)
    }
    nk := P256PublicKey{}

    err = json.Unmarshal(bz, &nk)
    if err != nil {
        panic(err)
    }

    newPub := key.Key()

    log.Printf("is equal: %v", newPub.Equal(pub))
}

For internal pkg change, maybe we can add a curve Identifier() method, then for Marshaler, we save the curve identifier + Point(X,Y). For UnMarshaler, we can recognize the public's curve from curve identifier, and Point(X,Y).