On the use of libsodium library,ED25519 algorithm signature error problem

[tebox]

Hello. I am getting a signature error when trying to call the /sell/finances/v1/transaction API. I am using the ED25519 algorithm with a private key obtained from the createSigningKey API.

Below are our signature data and the signature process. Could you please help me identify which step is causing the issue?

Thank you very much.

Below are the encrypted string and the key required for calling the /sell/finances/v1/transaction API.

message: "x-ebay-signature-key": eyJ6aXAiOiJERUYiLCJraWQiOiJiNmI4ZWY2MC0zODU4LTRiMGUtYTI5My1mZjQyOGJkZmMyZmMiLCJlbmMiOiJBMjU2R0NNIiwidGFnIjoiLW85eW5vTS1DTUllTVNnd2E1blpHdyIsImFsZyI6IkEyNTZHQ01LVyIsIml2IjoibDF0Q2VwLTFYVm5TUnI4ZSJ9.wV_-NGf8hv4W6i5l1FgPVLRBqWmCSg3VsH6P4jWEOW8.78NYWUx6oWTfKJOz.0PsXq8rLGouWGjIllEVTvPc0wXnb-aFk03w181B_nt6vfdOMfmSk-J0O6jecFsmFvkxmc726fkhhRFlM9hW3jgZz121JZW_qccfWmp3IhwAJjqmeqtazaPZpW5qDUlEPgXgVUTSQIgXgrkFzW-iKl7O4IjUVc5ae4-roKTGyV3dCCch7U6T_JQkHileEw4RDrkyiCGabDYqh0pgDFcxB-drLl-YbpKe54oJD2ClRJyxPPzKSD0a0dkMc-c0JFJBlTvY7uBoG.zQQaOyWA4F3IPqLn4HA_gg "@method": GET "@path": /sell/finances/v1/transaction "@authority": apiz.ebay.com "@signature-params": ("x-ebay-signature-key" "@method" "@path" "@authority");created=1684129818

PrivateKey: ......

Here is our encryption process:

1. Convert the private key and the string to be signed into UTF-8 encoded byte data. The following is the hexadecimal representation: message: 22 78 2D 65 62 61 79 2D 73 69 67 6E 61 74 75 72 65 2D 6B 65 79 22 3A 20 65 79 4A 36 61 58 41 69 4F 69 4A 45 52 55 59 69 4C 43 4A 72 61 57 51 69 4F 69 4A 69 4E 6D 49 34 5A 57 59 32 4D 43 30 7A 4F 44 55 34 4C 54 52 69 4D 47 55 74 59 54 49 35 4D 79 31 6D 5A 6A 51 79 4F 47 4A 6B 5A 6D ...omitted

PrivateKey: ...

2. We use the libsodium library for ED25519 encryption. For more information about the libsodium library, please refer to https://doc.libsodium.org/ The prototype of its export function is: SODIUM_EXPORT int crypto_sign_ed25519_detached(unsigned char sig, unsigned long long siglen_p, const unsigned char m, unsigned long long mlen, const unsigned char sk) attribute ((nonnull(1, 5)));

3. Call crypto_sign_ed25519_detached for signing:

if (crypto_sign_ed25519_detached(signature, &signatureLength, message, messageLength, privateKey) != 0) { std::cerr << "Signing failed." << std::endl; return 1; }

4. After successful signing, the signature buffer receives the following data: A1 6F 22 C5 27 1F F5 78 13 64 7B 39 F5 81 34 60 A2 42 FA 5F D9 FB 04 22 F1 4E 45 AC 67 51 F5 C3 A2 2F 0B 69 CD 9D BA E7 1B 88 A8 B5 8F A3 05 C6 DA 81 7E CB 70 0D 8E DD 70 F9 D5 B8 68 B6 A5 09

5. Base64 encode the data in the signature buffer, resulting in: oW8ixScf9XgTZHs59YE0YKJC+l/Z+wQi8U5FrGdR9cOiLwtpzZ265xuIq

6. The interface returns an error message indicating invalid signature verification { "errors":[ { "errorId":215122, "domain":"ACCESS", "category":"REQUEST", "message":"Signature validation failed", "longMessage":"Signature validation failed to fulfill the request." } ] }

[uherberg]

@tebox I couldn't find much documentation about the mentioned function crypto_sign_ed25519_detached. But inside the code, I see that they use SHA512. As per the IETF draft we follow, no hash is used. I recommend trying out the exact examples in the IETF draft (using the sample keys from the draft). That may help you to verify that your output is as expected.

[uherberg]

@tebox P.S. It's better to not post production private keys on github. If you need to, use a sandbox key.

[tebox]

@tebox I couldn't find much documentation about the mentioned function crypto_sign_ed25519_detached. But inside the code, I see that they use SHA512. As per the IETF draft we follow, no hash is used. I recommend trying out the exact examples in the IETF draft (using the sample keys from the draft). That may help you to verify that your output is as expected.

Hello, please check. https://www.rfc-editor.org/rfc/rfc8032#section-5.1.6, about the signature section description. SHA-512 is applied to Private_Key. "The signature base is taken as the input message () with no pre-hash function." described above should mean that the message to be signed is passed directly. The message does not need to be hashed before signing. In addition, I googled libsodium ED25519, which is standard for RFC 8032

Also about B.1.4. Example Ed25519 Test Key. The Private_Key and message body used is this right? Private_Key MC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF

The message body is (the original example has \ removed. Is that correct? "date": Tue, 20 Apr 2021 02:07:55 GMT "@method": POST "@path": /foo "@authority": example.com "content-type": application/json "content-length": 18 "@signature-params": ("date" "@method" "@path" "@authority" "content-type" "content-length"); created=1618884473; keyid="test-key-ed25519"

[uherberg]

@tebox Thanks for correcting me here. Yes, the above key and signature base look accurate. The message itself is shown in section B.2:

POST /foo?param=Value&Pet=dog HTTP/1.1
Host: example.com
Date: Tue, 20 Apr 2021 02:07:55 GMT
Content-Type: application/json
Content-Digest: sha-512=:WZDPaVn/7XgHaAy8pmojAkGWoRx2UFChF41A2svX+T\
  aPm+AbwAgBWnrIiYllu7BNNyealdVLvRwEmTHWXvJwew==:
Content-Length: 18

{"hello": "world"}

The output should be:

Signature-Input: sig-b26=("date" "@method" "@path" "@authority" \
  "content-type" "content-length");created=1618884473\
  ;keyid="test-key-ed25519"
Signature: sig-b26=:wqcAqbmYJ2ji2glfAMaRy4gruYYnx2nEFN2HN6jrnDnQCK1\
  u02Gb04v9EDgwUPiu4A0w6vuQv5lIp5WPpBKRCw==:

[tebox]

Private_Key MC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF

hello Since the libsodium library is suitable. Then I'd like to double check your signature. For example, Private_Key uses: MC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF

Signed message: 123456789

We each sign once. How about we check the signatures?

[uherberg]

@tebox Similar to what our PHP SDK does, I created this example:

<?php
use phpseclib3\Crypt\PublicKeyLoader;
require 'vendor/autoload.php';

$message = "123456789";
$privateKeyStr = "-----BEGIN PRIVATE KEY-----\nMC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF\n-----END PRIVATE KEY-----";
$private = PublicKeyLoader::loadPrivateKey($privateKeyStr);
$signed = $private->sign($message);
echo base64_encode($signed);

The result is:

W+IJiGXD8hgQnjqH9nVg6+hJAU7NuNem4EC4w7ly1+dMRuJXpvGv0Vg+Rm3c5hQTs7tR77yxvzpgcvaZOn//DQ==

[uherberg]

Have you checked that your code creates the same signature as in the IETF draft?

[tebox]

Have you checked that your code creates the same signature as in the IETF draft?

MC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF

123456789

result: rvw/tomN4jxtdWDJZltlTgyoxh57J1VapopUxYM/GqO9wNCXO685+SrzQkSZ0dXhtzHslHWgbt3p'#$D#$A'56T+KRNwCA==

hello The signatures do make a difference. It looks like the libsodium and PHP SDK signing processes process data differently. I don't know anything about algorithms. I wonder if you could help me look for different places. How can I modify it? Thank you very much

[uherberg]

@tebox I updated the example above to use just plain phpseclib3 and nothing of our SDK. I remember that I initially used libsodium for php, and struggled as well to recreate the signatures from the Internet Draft. That's when I switched to phpseclib3. I will see if I can play around with libsodium a bit more. Alternatively, maybe you could consider another library?

[tebox]

@tebox I updated the example above to use just plain phpseclib3 and nothing of our SDK. I remember that I initially used libsodium for php, and struggled as well to recreate the signatures from the Internet Draft. That's when I switched to phpseclib3. I will see if I can play around with libsodium a bit more. Alternatively, maybe you could consider another library?

Hello, because I use Delphi development. However Delphi does not have source code support ED25519 signature algorithm. So now I can only find DLL to call. This allows you to use libraries developed in C/C++. So I can't use the JAVA/PHP SDK either. Too bad I searched a lot of libraries, like Botan, etc. Most of them only have C++ source code. Cannot be used on DELPHI libsodium is the only library found that supports DLL calls. So I had no choice. If I hope you can help me to see how to solve this problem. Thank you so much!!

[uherberg]

@tebox Okay, I will play with it. How do I load the private key from the PEM file using libsodium?

[tebox]

@tebox Okay, I will play with it. How do I load the private key from the PEM file using libsodium?

Use libsodium without loading Private_Key from the PEM file. Simply read the Private_Key string into the byte. Other PEM file inside (-- -- -- -- -- BEGIN PRIVATE KEY -- -- -- -- --) and END (-- - END PRIVATE KEY -- -- -- -- --) string does not need Here is an example of C++

include

include

int main() { if (sodium_init() < 0) { std::cerr << "libsodium initialization failed." << std::endl; return 1; }

// Generate a public and private key pair // Here you can simply replace it with the Private_Key of the test MC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF

unsigned char publicKey[crypto_sign_PUBLICKEYBYTES]; unsigned char privateKey[crypto_sign_SECRETKEYBYTES]; crypto_sign_keypair(publicKey, privateKey);

// Prepare the message for signature const unsigned char message = (const unsigned char)"Hello, World!" ; const size_t messageLength = 13;

// Allocate the signature buffer unsigned char signature[crypto_sign_BYTES]; unsigned long long signatureLength;

// Sign the file if (crypto_sign_ed25519_detached(signature, &signatureLength, message, messageLength, privateKey) ! = 0) { std::cerr << "Signing failed." << std::endl; return 1; }

// Print the signature result std::cout << "Signature: "; for (unsigned int i = 0; i < signatureLength; ++i) { std::cout << std::hex << std::setw(2) << std::setfill('0') << static_cast(signature[i]); } std::cout << std::endl;

return 0; }

[tebox]

@tebox Okay, I will play with it. How do I load the private key from the PEM file using libsodium?

And this is Delphi calling code. Simply pass in the signature string and key. The above C++ code is the same, simply call the function I don't know if it'll help you.

  Private_Key := 'MC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF';
  SigStr := '123456789';
  Ed25519Sign( Private_Key , SigStr );

Function Ed25519Sign( PrivateKeyStr , SinStr : String ) : String;
Var
  Messages , PrivateKey , Signature : TBytes;
  Siglen : UInt64;
Begin
  Result := '';

  Siglen := 256;
  SetLength( Signature , Siglen );
  ZeroMemory( @Signature[0] , Siglen );

  Messages := TEncoding.UTF8.GetBytes( SinStr );
  PrivateKey := TEncoding.UTF8.GetBytes( PrivateKeyStr );
  // PrivateKey := TNetEncoding.Base64.DecodeStringToBytes( PrivateKeyStr );

  // 进行签名
  If Crypto_sign_ed25519_detached( @Signature[0] , Siglen , @Messages[0] , Length( Messages ) , @PrivateKey[0] ) = 0 Then
  Begin
    // 更新实际使用的签名长度
    SetLength( Signature , Siglen );

    // base64
    Result := TNetEncoding.Base64.EncodeBytesToString( Signature );
  End;

End;

[uherberg]

@tebox I suspect the issues is with loading the keys. The example in C above creates a new random key. That's all I found also in their documentation. I am not sure how to load the secret key with sodium; just loading the key from the PEM can't be it (that is 48 bytes long, but sodium apparently expects 64 bytes). Your code seems to convert the base64 encoded PEM into UTF8 bytes; that would create a different key.

[tebox]

@tebox I suspect the issues is with loading the keys. The example in C above creates a new random key. That's all I found also in their documentation. I am not sure how to load the secret key with sodium; just loading the key from the PEM can't be it (that is 48 bytes long, but sodium apparently expects 64 bytes). Your code seems to convert the base64 encoded PEM into UTF8 bytes; that would create a different key.

Do you mean that it is wrong for me to take the 64-bit key and put it in byte directly? PrivateKey = TEncoding.UTF8.GetBytes( "MC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF")

base64 decrypted 48 bit key is correct? PrivateKey = TNetEncoding.Base64.DecodeStringToBytes("MC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF")

[uherberg]

@tebox Yes, that wouldn't work. A PEM file contains a byte64 encoded PKCS8 key (a 48 byte long ASN.1 structure that includes the 32-byte long key and an algorithm and version). libsodium apparently uses a proprietary key format of 64 byte. Not sure yet if one can convert ASN.1 to this structure, as they didn't document it.

[tebox]

@tebox I suspect the issues is with loading the keys. The example in C above creates a new random key. That's all I found also in their documentation. I am not sure how to load the secret key with sodium; just loading the key from the PEM can't be it (that is 48 bytes long, but sodium apparently expects 64 bytes). Your code seems to convert the base64 encoded PEM into UTF8 bytes; that would create a different key.

In addition, I don't know PHP. I searched for the PHP code that calls the libsodium library. I don't know if it'll help you As shown in the code. A 64-bit key is used directly to sign. Maybe it's the ASN.1 structure key you're talking about that's causing the problem?

// 导入libsodium库
if (!extension_loaded('sodium')) {
    dl('sodium.so');
}

// 签名并进行Base64编码
function signMessage($message, $secretKey) {
    $signature = sodium_crypto_sign_detached($message, $secretKey);
    $encodedSignature = base64_encode($signature);
    return $encodedSignature;
}

// 示例用法
$message = "123456789";
$secretKey = hex2bin('MC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF');

$encodedSignature = signMessage($message, $secretKey);

echo "Message: " . $message . "\n";
echo "Encoded Signature: " . $encodedSignature . "\n";

[uherberg]

@tebox PHP just has a wrapper to libsodium; it would be the same from Delphi. hex2bin will not work; this is Base64 encoded ASN.1. sodium_crypto_sign_detached (or crypto_sign_detached in C) expects this 64 byte key format from libsodium, not a 48 byte ASN.1 structure (or the 32 byte raw key). Maybe you can explore other libraries?

[tebox]

@tebox PHP just has a wrapper to libsodium; it would be the same from Delphi. hex2bin will not work; this is Base64 encoded ASN.1. sodium_crypto_sign_detached (or crypto_sign_detached in C) expects this 64 byte key format from libsodium, not a 48 byte ASN.1 structure (or the 32 byte raw key). Maybe you can explore other libraries?

It's really a pity that the current public library does not support the ED25519 signature algorithm in the format of ANS.1 Only the original key is used for signature. I guess I'll have to give up on ED25519. I next tried signing using RSASSA_PKCS1_v1_5.

In addition, the RSA key. Is it also ASN.1 format? Or a regular key?

Let's go back to the previous data. Can you provide a signature result for my reference?

For example, Private_Key uses: MC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF

Signed message: 123456789

[uherberg]

@tebox You you'd like to get a sample signed with RSA? (the above key is still ED25519). RSA keys are much longer. Here is the code (using the sample key included in this repo):

<?php
use phpseclib3\Crypt\PublicKeyLoader;
require 'vendor/autoload.php';

$message = "123456789";
$privateKeyStr = file_get_contents("./keys/rsa/privatekey.pem");
$private = PublicKeyLoader::loadPrivateKey($privateKeyStr);
$signed = $private->sign($message);
echo base64_encode($signed);

The output is

EhhTKTBnNx4qIIt4gBP54Io02reMEEoyucUDksR7kA7dJddlv0NINw+gyxrrHhPOuydwCityAq6BFebg1DISDWCZtBoZ4ZK+4VyBP5VHTPl2xnNM4Zoj8Yq7NlJId54XmWoVvpyIEI1C6houKC588Jj4qdIB7B7tcKFki0JIFg+r5t+1W9SJtJgkW9vCTRgF0L07rxK9hLj3AYj59WXpescK9n3aNnubG3HT6LwTcg/B6IZj06gAZuaeU1vC/DMYj2NMbN+ryo2hd/F4KrxLYX1+bqVUNPltInc/l3mTeCtEtb7fTe8WJ2j48a+IAemQs5v+bN7+KVdbm2k+FHhi1g==

[tebox]

@tebox You you'd like to get a sample signed with RSA? (the above key is still ED25519). RSA keys are much longer. Here is the code (using the sample key included in this repo):

<?php
use phpseclib3\Crypt\PublicKeyLoader;
require 'vendor/autoload.php';

$message = "123456789";
$privateKeyStr = file_get_contents("./keys/rsa/privatekey.pem");
$private = PublicKeyLoader::loadPrivateKey($privateKeyStr);
$signed = $private->sign($message);
echo base64_encode($signed);

The output is

EhhTKTBnNx4qIIt4gBP54Io02reMEEoyucUDksR7kA7dJddlv0NINw+gyxrrHhPOuydwCityAq6BFebg1DISDWCZtBoZ4ZK+4VyBP5VHTPl2xnNM4Zoj8Yq7NlJId54XmWoVvpyIEI1C6houKC588Jj4qdIB7B7tcKFki0JIFg+r5t+1W9SJtJgkW9vCTRgF0L07rxK9hLj3AYj59WXpescK9n3aNnubG3HT6LwTcg/B6IZj06gAZuaeU1vC/DMYj2NMbN+ryo2hd/F4KrxLYX1+bqVUNPltInc/l3mTeCtEtb7fTe8WJ2j48a+IAemQs5v+bN7+KVdbm2k+FHhi1g==

1. In addition, the RSA key. Is it also ASN.1 format? Or a regular key?

2. Can you send the contents of the privatekey.pem file? I'll run a test on your key here

[uherberg]

1. Also PEM format. That's the standard everywhere (besides apparently libsodium)
2. https://github.com/eBay/digital-signature-verification-ebay-api/blob/main/src/main/resources/keys/rsa/privatekey.pem

[tebox]

1. Also PEM format. That's the standard everywhere (besides apparently libsodium)
2. https://github.com/eBay/digital-signature-verification-ebay-api/blob/main/src/main/resources/keys/rsa/privatekey.pem

Thank you very much. I will try it and contact you if I have any problems. The problem so far seems to be the key structure of ASN.1. Thank you again for your enthusiasm. thank you

[tebox]

1. Also PEM format. That's the standard everywhere (besides apparently libsodium)
2. https://github.com/eBay/digital-signature-verification-ebay-api/blob/main/src/main/resources/keys/rsa/privatekey.pem

hello I've tried a lot of things in the last two days. Still cannot implement the signing process. May I ask if you can compile the code of the signature part into DLL? I can call the export function of DLL for message signature.ED25519 or RSA will do. That's a valid and reliable solution, right?

[uherberg]

@tebox I think I found a solution. Basically, convert the key to DER (base64_decode the key), then take the substring of byte 16 to 48, and use that as seed to create the key. Here a sample code in PHP (which should be easy to rewrite in Delphi)

<?php
use phpseclib3\Crypt\PublicKeyLoader;
require 'vendor/autoload.php';

$message = "123456789";

// First with phpseclib3
$privateKeyStr = "-----BEGIN PRIVATE KEY-----\nMC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF\n-----END PRIVATE KEY-----";
$private = PublicKeyLoader::loadPrivateKey($privateKeyStr);
$signed = $private->sign($message);
echo "phpseclib3: " . base64_encode($signed) . "\n";

// Now to compare with libsodium
$privateKeyStr = "MC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF";
$private = base64_decode($privateKeyStr, true);
$private = substr($private, 16, 48);
$keypair = sodium_crypto_sign_seed_keypair($private);
$secret = sodium_crypto_sign_secretkey($keypair);
$signed = sodium_crypto_sign_detached($message, $secret);
echo "sodium:     " . base64_encode($signed)  . "\n";

Output is:

phpseclib3: W+IJiGXD8hgQnjqH9nVg6+hJAU7NuNem4EC4w7ly1+dMRuJXpvGv0Vg+Rm3c5hQTs7tR77yxvzpgcvaZOn//DQ==
sodium:     W+IJiGXD8hgQnjqH9nVg6+hJAU7NuNem4EC4w7ly1+dMRuJXpvGv0Vg+Rm3c5hQTs7tR77yxvzpgcvaZOn//DQ==

[tebox]

@tebox I think I found a solution. Basically, convert the key to DER (base64_decode the key), then take the substring of byte 16 to 48, and use that as seed to create the key. Here a sample code in PHP (which should be easy to rewrite in Delphi)

<?php
use phpseclib3\Crypt\PublicKeyLoader;
require 'vendor/autoload.php';

$message = "123456789";

// First with phpseclib3
$privateKeyStr = "-----BEGIN PRIVATE KEY-----\nMC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF\n-----END PRIVATE KEY-----";
$private = PublicKeyLoader::loadPrivateKey($privateKeyStr);
$signed = $private->sign($message);
echo "phpseclib3: " . base64_encode($signed) . "\n";

// Now to compare with libsodium
$privateKeyStr = "MC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF";
$private = base64_decode($privateKeyStr, true);
$private = substr($private, 16, 48);
$keypair = sodium_crypto_sign_seed_keypair($private);
$secret = sodium_crypto_sign_secretkey($keypair);
$signed = sodium_crypto_sign_detached($message, $secret);
echo "sodium:     " . base64_encode($signed)  . "\n";

Output is:

phpseclib3: W+IJiGXD8hgQnjqH9nVg6+hJAU7NuNem4EC4w7ly1+dMRuJXpvGv0Vg+Rm3c5hQTs7tR77yxvzpgcvaZOn//DQ==
sodium:     W+IJiGXD8hgQnjqH9nVg6+hJAU7NuNem4EC4w7ly1+dMRuJXpvGv0Vg+Rm3c5hQTs7tR77yxvzpgcvaZOn//DQ==

This is a very good idea. I'll try. Thanks

[uherberg]

Closing this issue, as there was no problem with our code