Signature mismatches between misc. secp256k1 libraries, secp256k1-node is the odd man out

[djvs]

Everything below meant to indicate a hex encoding of a Buffer or Uint8Array.

For the private key: a745374f134bdc1c1197acbe6b7dc6ed16c3ea65f8606cee7f0bf80c2df8540f

With corresponding uncompressed pubkey: (04) e6fdf03cdc74c0f4f2208198adb5795ecfa82ad3ab13f4a00429106c10e0370c89da77a39617f3d76c015a9ea761593ec35c533ac69d861e81a25a86c6116f5e

Signing this 32 byte hash: ea095d83ff149caddeade14eb838ddbc76ac62b7ec449a03e2c4b28ed4528641

Every other library I tested (all based on libsecp256k1, some Swift, some Ruby, etc.) - with the single exception of ethereum-cryptography - all return something equivalent to this: ea6f005f77c65e0a76231c00afab03ef1eed7eb56d8a3ee7f972804f20d85beb1df3fd34e15f78f3aacdd963079e8c5526f4d4047083c64bf54cdf45e2777475

However the node library returns this: eb5bd8204f8072f9e73e8a6db57eed1eef03abaf001c23760a5ec6775f006fea757477e245df4cf54bc6837004d4f426558c9e0763d9cdaaf3785fe134fdf31d

It doesn't appear to be normalized signatures. I don't think it's the nonce function. I'm kind of out of ideas and worn out from investigating.

Sample code:

s = require('secp256k1')

privkey = Buffer.from('a745374f134bdc1c1197acbe6b7dc6ed16c3ea65f8606cee7f0bf80c2df8540f', 'hex')
msghash = Buffer.from('ea095d83ff149caddeade14eb838ddbc76ac62b7ec449a03e2c4b28ed4528641', 'hex')
sig = s.ecdsaSign(msghash, privkey)
console.log({
  msghash: msghash.toString('hex'),
  privkey: privkey.toString('hex'),
  sig: Buffer.from(sig.signature).toString('hex'),
})

/* returns:
{
  msghash: 'ea095d83ff149caddeade14eb838ddbc76ac62b7ec449a03e2c4b28ed4528641',
  privkey: 'a745374f134bdc1c1197acbe6b7dc6ed16c3ea65f8606cee7f0bf80c2df8540f',
  sig: 'eb5bd8204f8072f9e73e8a6db57eed1eef03abaf001c23760a5ec6775f006fea757477e245df4cf54bc6837004d4f426558c9e0763d9cdaaf3785fe134fdf31d'
}
*/

Ruby:

# gem install secp256k1-ruby   

require 'secp256k1'  
privkeystr  = "a745374f134bdc1c1197acbe6b7dc6ed16c3ea65f8606cee7f0bf80c2df8540f".scan(/../).inject(""){|b,hn|b<<hn.to_i(16).chr}  

privkey = Secp256k1::PrivateKey.new(privkey: privkeystr )  

msg = "waste library general honey shop hockey easily resist miracle injury depend lunar income thrive raise height eight lab rack state armed acid lady soccer" # hashes to "ea095d83ff149caddeade14eb838ddbc76ac62b7ec449a03e2c4b28ed4528641"

sig = privkey.ecdsa_sign(msg)   

sig.read_string_length(64).split("").map{|x|x.ord.to_s(16)}.join('')

# returns: "ea6f05f77c65ea76231c0afab3ef1eed7eb56d8a3ee7f972804f20d85beb1df3fd34e15f78f3aacdd96379e8c5526f4d447083c64bf54cdf45e2777475"

I don't know if this is a bug or just difference in invocation. I would really appreciate some clarification.

[jklein24]

@djvs did you ever figure this out? I'm seeing a similar situation right now where secp256k1 libraries in other languages are returning a different result than this one.

[djvs]

@jklein24 I did not, no.

[junderw]

There's no benefit or requirement for ECDSA signatures from various libraries to always be exactly the same.

This library uses RFC6979, which is deterministic, but it allows you do include extra entropy to change the deterministic signature output when needed.

A lot of times, these sorts of minor differences are born of semantics of different languages.

ie. the C library might take a null pointer to mean "don't perform the extra entropy hashing round" whereas Ruby or some other language will take a null value to mean "hash the extra entropy round with a 32 byte array of 0x00 bytes." etc.

Investigating the cause of these minor differences requires expertise in low level aspects of each of these languages.

tl;dr as long as the library is outputing a valid signature, and it is deterministic using RFC6979, you are fine. There is no benefit to matching exact signature outputs.

If there is some sort of protocol that requires deterministic signature outputs to be guaranteed between languages, then that protocol is poorly designed.

[junderw]

This library is just a JS wrapper around the C library, and the output matches the C library at the version we target.

(Note: This schorr part is unrelated to this wrapper library in particular, but is relevant to another library I maintain)

One notable update in the C library had schnorr signatures change due to a different interpretation of the schorrsig BIP and how the "auxilary data" (similar to "extra entropy" in RFC6979) is treated.

This caused a stir. I get people are touchy about signatures and think "the only way to ensure we are generating RFC6979 signatures properly is to run tests against known-good RFC6979 signatures so they never change"...

The reality is, no, that's not a good way to judge it.

If someone who had access wanted to put a backdoor in a signature library, they could just do a survey of all downstream projects that test signatures and put in special cases for those specific signatures.

When you update your dependency, validate the diff and ensure it's good. That's the only way.

A signature test might be a good way to recognize some sort of change going on, but viewing that as a "bug" or a "breaking change" of the crypto library is incorrect.

[junderw]

Some things I noted after a cursory glance:

secp256k1-ruby last update was almost 5 years ago and is a Ruby wrapper around the same library we use, but the commit it wraps is over 5 years old.

We target a commit that's about 4 years old. So maybe something happened in the C library during that year, OR some semantic thing about how Ruby or Ruby FFI works changed the output.

[junderw]

https://github.com/bitcoin-core/secp256k1/compare/e34ceb333b1c0e6f4115ecbb80c632ac1042fa49...d644dda5c9dbdecee52d1aa259235510fdc2d4ee

Looking at the diff of the commits we both target, the biggest difference was the addition of precompute.

The nonce function and everything else is untouched, it seems.

Probably something to do with Ruby or Ruby's FFI library.

[junderw]

I can also verify that tiny-secp256k1 also generates the same signature as this library. (It is also based on libsecp256k1)

[junderw]

The Ruby output you posted is 61 bytes. A valid signature is 64 bytes...

This seems like incorrect usage of the Ruby library.

[djvs]

Thanks a lot for looking into it @junderw, appreciate it