ShadowDecrypt method conversion to JS

[ArvsIndrarys]

• Parity Ethereum version: 2.3.0-unstable-ac974a1-20181114
• Operating system: Linux Ubuntu 16.04
• Installation: built from source with secret-store (cargo build --features secretstore --release)
• Fully synchronized: yes
• Network: dev
• Restarted: yes

Hi everyone,
my goal would be to interact with the Secret Store via JS. The environment of my setup would allow me to trust no-one, thus having all plain datas decrypted only on the Client Side.
I am using the Secret Store feature of document key shadow retrieval and avoid using the shadow_decrypt rpc call to an untrusteed node.

I would then need to implement that call in JS.But I don't understand one of the function calls :

( original code is here )

struct ByteBuf<'a>(&'a [u8]);

impl<'a> std::fmt::LowerHex for ByteBuf<'a> {
    fn fmt(&self, fmtr: &mut std::fmt::Formatter) -> Result<(), std::fmt::Error> {
        for byte in self.0 {
            try!( fmtr.write_fmt(format_args!("{:02x}", byte)));
        }
        Ok(())
    }
}

fn to_secp256k1_public(public: &Public) -> Result<key::PublicKey, Error> {

    println!("PUBLIC BEFORE MODIFICATION: {:?}", public);
    // Creates a new array of size 65 and concatenates '04' with public
    let public_data = {
        let mut temp = [4u8; 65];
        (&mut temp[1..65]).copy_from_slice(&public[0..64]);
        temp
    };

    println!("JUST ADDED 04 BEFORE THE PUBKEY? : {:x}", ByteBuf(&public_data));

    println!("PUBLIC TO PUBLIC KEY: {:?}", key::PublicKey::from_slice(&SECP256K1, &public_data)?);

    Ok(key::PublicKey::from_slice(&SECP256K1, &public_data)?)
}

The struct above the to_secp256k1_public function is from StackOverflow and helps me debug the content of the array before it is modified by the key::PublicKey::from_slice call.

I re-adapted this code in JS thanks to that part :

    console.log("COMMON POINT: ", commonPoint);
    const commonPointBuffer = Buffer.concat([Buffer.from('04', 'hex'), string2HexBuffer(commonPoint)]);

    console.log("JUST 04 added before the common point?", commonPointBuffer.toString('hex'));

    const convertedCommonPointBuffer = secp256k1.publicKeyConvert(commonPointBuffer, false);
    console.log('CONVERTED?: ', convertedCommonPointBuffer.toString('hex'));

I get then these results :

[JS]

COMMON POINT:  0x1cdd523dc2a4a81a6801504d60a00214e58248224ad075f32035d8dbc0a1602772162b5d1fad051e64f941aa9f930d33a76982d08c8bf0e7a4b36062d6edb48b
JUST 04 added before the common point? 041cdd523dc2a4a81a6801504d60a00214e58248224ad075f32035d8dbc0a1602772162b5d1fad051e64f941aa9f930d33a76982d08c8bf0e7a4b36062d6edb48b
CONVERTED?:  041cdd523dc2a4a81a6801504d60a00214e58248224ad075f32035d8dbc0a1602772162b5d1fad051e64f941aa9f930d33a76982d08c8bf0e7a4b36062d6edb48b

[Parity]

PUBLIC BEFORE MODIFICATION: 0x1cdd523dc2a4a81a6801504d60a00214e58248224ad075f32035d8dbc0a1602772162b5d1fad051e64f941aa9f930d33a76982d08c8bf0e7a4b36062d6edb48b
JUST ADDED 04 BEFORE THE PUBKEY? : 041cdd523dc2a4a81a6801504d60a00214e58248224ad075f32035d8dbc0a1602772162b5d1fad051e64f941aa9f930d33a76982d08c8bf0e7a4b36062d6edb48b
PUBLIC TO PUBLIC KEY: PublicKey(2760a1c0dbd83520f375d04a224882e51402a0604d5001681aa8a4c23d52dd1c8bb4edd66260b3a4e7f08b8cd08269a7330d939faa41f9641e05ad1f5d2b1672)

For Parity, the call key::PublicKey::from_slice(&SECP256K1, &public_data) comes from the parity-secp256k1 rust wrapper and calls the underlying secp246k1 C implementation from that code :

    /// Creates a public key directly from a slice
    #[inline]
    pub fn from_slice(secp: &Secp256k1, data: &[u8])
                      -> Result<PublicKey, Error> {

        let mut pk = unsafe { ffi::PublicKey::blank() };
        unsafe {
            if ffi::secp256k1_ec_pubkey_parse(secp.ctx, &mut pk, data.as_ptr(),
                                              data.len() as usize) == 1 {
                Ok(PublicKey(pk))
            } else {
                Err(InvalidPublicKey)
            }
        }
    }

For the JS, I am using the secp256k1-node wrapper that calls the underlying secp246k1 C implementation from that code :

    publicKeyConvert: function (publicKey, compressed) {
      assert.isBuffer(publicKey, messages.EC_PUBLIC_KEY_TYPE_INVALID)
      assert.isBufferLength2(publicKey, 33, 65, messages.EC_PUBLIC_KEY_LENGTH_INVALID)

      compressed = initCompressedValue(compressed, true)

      return secp256k1.publicKeyConvert(publicKey, compressed)
    },

I don't understand how Parity returns a public key THAT different from the entry, as it is already a correct public key, and the method called is publicKeyConvert.
And how could I obtain the same result (2760a1c0dbd83520f375d04a224882e51402a0604d5001681aa8a4c23d52dd1c8bb4edd66260b3a4e7f08b8cd08269a7330d939faa41f9641e05ad1f5d2b1672) in JS?

[jam10o-new]

I'm not very familiar with these - but my first thought is that publicKeyConvert should be analogous to me serialize_vec on the rust side, not from_slice

I'd wait for someone more familiar with the secret store to respond though (cc @grbIzl)

[ArvsIndrarys]

The thing is, Parity returns the correct Public Key that allows the Document key recuperation, which I can not do in JS. So either Parity wraps correctly the original secp256k1 C implementation and the others, not ( ethers-io/ethers.js for example), either that part is flawed and the code has been adapted to get the correct result from that flaw (perhaps the same way).

[ArvsIndrarys]

I don't expect the difference between serialize_vec and from_slice to be the problem. I got no problem with secretKey.to_secp256k1_secret() that uses from_slice. I am adapting the full rpc shadow decrypt code and I got no problem until the key::PublicKey::from_slice() method. Only that part gives me a different result than the bitcoin secp256k1 implementation.

[svyatonik]

The public keys looks the same, though coordinates are serialized differently. Public key is simply 2x serialized 256-bit numbers (x and y coordinates of a point on the SECP256k1 curve) and the 04 is simply tag that points to serialization format. Please mention that coordinates in JS and Parity are simply reversed. I.e. in JS x coordinate is 1cdd523dc2a4a81a6801504d60a00214e58248224ad075f32035d8dbc0a16027 and in Parity it is reverse-bytes(1cdd523dc2a4a81a6801504d60a00214e58248224ad075f32035d8dbc0a16027) = 2760a1c0dbd83520f375d04a224882e51402a0604d5001681aa8a4c23d52dd1c. So either both libraries are compatible but are printing keys differently, or JS library expects some different format of the public key in its parse(). You could try doing some math with your JS keys (like scalar * pub_key) && check if result is the same (with regard to reverse-bytes()) as in rust - if they're different, try parsing/creating public key in JS with reversing + probably without prefix tag.

[svyatonik]

Please reopen if you need more help. And thanks for your interest in secret store

[ArvsIndrarys]

I did not see the reverse_bytes thingy ! My solution is working now, thanks @svyatonik !