Incorrect XML hashing

[HaimenToshi]

Hello. Thank you for this detailed project. I've been having issues with Phase 2 QR, getting the following errors in ZATCA's XML Web validator :

category : QR_CODE_ERROR
code :publicKey
message : public key of the certificate does not match with qr code certificate

category : QR_CODE_ERROR
code :hashedXml
message : hashedXml does not match with qr code hashedXml

category : QR_CODE_ERROR
code :certificate signature
message : certificate signature value in the invoice doesn't match the certificate signature value in tag 9 of the QR code

---

Let's take the hashedXml error first : I believe that ZATCA requires the XML to be hashed into SHA256 binary object before converting to base64, however the function "getInvoiceHash(invoice_xml)" takes the plain SHA256 and convert directly to base64. could that be the reason for the HashedXml error ?

[wes4m]

@HaimenToshi Are you testing the generated XML using their web validator? if so then the errors are expected. ZATCA's web validator uses hard-coded certs and keys. If using the Java SDK then you have to replace the certs and keys in the data folder. Otherwise just test using the API and supply the correct certs.

To confirm use the example included in this lib. If the compliance API request passes then the XML is correctly signed.

[HaimenToshi]

@wes4m Thanks for taking the time to look into this matter. When exactly do I use the getInvoiceHash function? Currently, I'm using it after building the barebones invoice structure, just the store info and items. Is it the correct time to hash the XML? or should I sign the XML first, then hash the XML and generate the QR in the end?

[wes4m]

@HaimenToshi the signing happens after getting the XML hash. since the digital signature is basically the invoice hash signed using the private key. I need more details/code to help you.

[HaimenToshi]

@wes4m Basically I construct the XML string, then use the getPureInvoiceString + getInvoiceHash functions on it :

EDIT : see attached file, github misses the code badly. zat.js.txt

This is the whole process / code I'm using right now. Using this method the web validator only complains about the 3 QR code fields I originally posted. The compliance API however refuses the document completely stating that "The invoice has invalid fields".

[HaimenToshi]

UPDATE : Finally get the invoice API to respond to my requests, here is the reporting results :

{ validationResults: { infoMessages: [ { type: "INFO", code: "XSD_ZATCA_VALID", category: "XSD validation", message: "Complied with UBL 2.1 standards in line with ZATCA specifications", status: "PASS" } ], warningMessages: [], errorMessages: [ { type: "ERROR", code: "invalid-invoice-hash", category: "INVOICE_HASHING_ERRORS", message: "The invoice hash API body does not match the (calculated) Hash of the XML", status: "ERROR" }, { type: "ERROR", code: "QRCODE_INVALID", category: "QRCODE_VALIDATION", message: "Unparsable QR Code", status: "ERROR" } ], status: "ERROR" }, reportingStatus: "NOT_REPORTED", clearanceStatus: null, qrSellertStatus: null, qrBuyertStatus: null }

EDIT : Interesting enough, changing the store name from Arabic to English was enough to get rid of QR parsing errors. Here is the updated response :

{ validationResults: { infoMessages: [ { type: "INFO", code: "XSD_ZATCA_VALID", category: "XSD validation", message: "Complied with UBL 2.1 standards in line with ZATCA specifications", status: "PASS" } ], warningMessages: [], errorMessages: [ { type: "ERROR", code: "invalid-invoice-hash", category: "INVOICE_HASHING_ERRORS", message: "The invoice hash API body does not match the (calculated) Hash of the XML", status: "ERROR" }, { type: "ERROR", code: "publicKey_QRCODE_INVALID", category: "QRCODE_VALIDATION", message: "ECDSA Public Key does not match with qr code ECDSA public key", status: "ERROR" }, { type: "ERROR", code: "CERTIFICATE_SIGNATURE_QRCODE_INVALID", category: "QRCODE_VALIDATION", message: "certificate signature does not match with qr certificate signature value ", status: "ERROR" } ], status: "ERROR" }, reportingStatus: "NOT_REPORTED", clearanceStatus: null, qrSellertStatus: null, qrBuyertStatus: null }

EDIT 2 : Got rid of the QR certificate signature error! :

{ validationResults: { infoMessages: [ { type: "INFO", code: "XSD_ZATCA_VALID", category: "XSD validation", message: "Complied with UBL 2.1 standards in line with ZATCA specifications", status: "PASS" } ], warningMessages: [], errorMessages: [ { type: "ERROR", code: "invalid-invoice-hash", category: "INVOICE_HASHING_ERRORS", message: "The invoice hash API body does not match the (calculated) Hash of the XML", status: "ERROR" }, { type: "ERROR", code: "publicKey_QRCODE_INVALID", category: "QRCODE_VALIDATION", message: "ECDSA Public Key does not match with qr code ECDSA public key", status: "ERROR" } ], status: "ERROR" }, reportingStatus: "NOT_REPORTED", clearanceStatus: null, qrSellertStatus: null, qrBuyertStatus: null }

@wes4m EDiT 3 : It seems that the QR Public key error stems from the fact that I have the public key in a separate file and not inside the x509 certificate itself. However, when I issued the certificate using ZATCA's API it didn't include the public key inside the issued certificate. Any insights on that would be really appreciated.

[asim009]

{ type: "ERROR", code: "invalid-invoice-hash", category: "INVOICE_HASHING_ERRORS", message: "The invoice hash API body does not match the (calculated) Hash of the XML", status: "ERROR" },

I have read somewhere in zatca document that for the very first time the invoice hash should be this "NWZlY2ViNjZmZmM4NmYzOGQ5NTI3ODZjNmQ2OTZjNzljMmRiYzIzOWRkNGU5MWI0NjcyOWQ3M2EyN2ZiNTdlOQ=="

[AhmedElywa]

@asim009

{ type: "ERROR", code: "invalid-invoice-hash", category: "INVOICE_HASHING_ERRORS", message: "The invoice hash API body does not match the (calculated) Hash of the XML", status: "ERROR" },

I have read somewhere in zatca document that for the very first time the invoice hash should be this "NWZlY2ViNjZmZmM4NmYzOGQ5NTI3ODZjNmQ2OTZjNzljMmRiYzIzOWRkNGU5MWI0NjcyOWQ3M2EyN2ZiNTdlOQ=="

Yes, you are right here https://zatca.gov.sa/ar/E-Invoicing/SystemsDevelopers/Documents/20220624_ZATCA_Electronic_Invoice_XML_Implementation_Standard_vF.pdf page 44

[HaimenToshi]

Thank you both, I managed to solve the private key issue (found the key in the certificate despite the fact that console.log refused to show it). And now I'm left with the XML hash issue alone. However, the error is not about the Previous Invoice Hash, but rather the Invoice Hash of the current XML that I'm sending to the API.

EDIT : Of course the XML hash error is not related to QR code, maybe you guys are correct, however setting the PIH to the fixed value from ZATCA results in the same error :

type: "ERROR", code: "invalid-invoice-hash", category: "INVOICE_HASHING_ERRORS", message: "The invoice hash API body does not match the (calculated) Hash of the XML"

As I understand it, it complains that the POST request body's 'invoiceHash' is not the same as the barebones XML's hash I generate before the signing process, which is nonsense as I put the same hash in the POST request. Any ideas? I'm completely lost.

[wes4m]

thanks everyone for the discussion. @HaimenToshi for your last error, I struggled for a while trying to get the API to get a match for the XML hash. They strip the XML from the signing props and rehash it and check the hash they get against the hash you provide. For those two hashes to match an absolutely exact format of the XML must be provided including (white space, new lines, tabs .. etc) so what I would suggest is to sign the XML using your code, and also sign it using their SDK and diff the two. Find what exactly is the difference and try to match it.

[HaimenToshi]

@wes4m Thanks for the clarification! As usual, ZATCA's practices are awful so to speak. White spaces, tabs and line breaks shouldn't affect the parsing of the XML (they exist just to make the document human readable), and I used the script and hashed an invoice, then added a few new lines and random white spaces to the same document, hashed it, and the EXACT SAME hash was returned. But ZATCA's hashing method seems over zealous. Will try my best to figure it out and will keep you posted. Cheers.

[wes4m]

@HaimenToshi agreed. I tried talking to them recently suggesting that they minify the XML before hashing but unfortunately couldn't convince them.

[HaimenToshi]

UPDATE : Still nothing. Taking the example from the ZATCA API, there is NO WAY to generate t1vEblOtMk4E3+YFofL4rYw8ARqvuvA5aQYVw6wS3BA= Hash from this invoice : example.xml.txt

No matter how much I clean & canon it. This is borderline depressing. 8 )

EDIT : Holy Eff. I FOUND IT!! Finally got to generate the exact same hash in the example above. @wes4m was correct, ZATCA turns the file to binary before hashing it, resulting in a different hash EVEN IF ONE WHITE SPACE IS DIFFERENT. You will have to adapt their exact same example, be careful not to miss up any white spaces or new lines, then you'll get the hash they are expecting. The getInvoiceHash function is missing a few new lines, so please update this function this one (attached file because github misses the code up) :

333.js.txt

@wes4m please update this function, I don't know how to do a PR. :)

[wes4m]

@HaimenToshi 😆good job. I feel your pain. the getInvoiceHash works provided you use the zatca-xml-js lib to generate the XML and not use your own XML. I made it so that the templates being used in the generation all include the dumb whitespaces, etc ... You can test using the examples and will see that it matches ZATCA's API just fine. I don't plan to modify the getInvoiceHash to account for other use cases for now at least.

If there are no other issues please close this as complete, thanks.

[HaimenToshi]

@wes4m You are absolutely correct, there is no need to change the getInvoicehash function once I figured out how to build a "proper" XML string. Now there is a strange thing going on : My XML is getting reported successfully when I send it using the API, however when I use the web validator to parse the same file, it complains about 3 things : the x509 issuer, the x509 serial number and the SignedProperties hash being incorrect. Which one should I believe, the Web validator of the API? lol

[wes4m]

@HaimenToshi the web validator uses a constant (hardcoded) certs and keys on ZATCA's side that they use to resign and do their thing with the XML. So if you're using a different cert it won't get a match. If i'm remembering correctly the certs and keys being used in the web validator are the same ones provided in the Java SDK.

[HaimenToshi]

@wes4m Thank you for clarifying! and again thank you for this awesome detailed lib. Please allow me to close this issue. Thanks.

[asim009]

@wes4m Thank you for such a good library. I want to give you a treat let me know how..:-p

[jayaganeshk]

@HaimenToshi can you please share the XML how it looks before generating the hash of it?

[HaimenToshi]

@jayaganeshk Sorry for the late reply. It is best to build the XML manually from the ground up in a clean way, and don't rely on the canonlizer & purifier functions. Or let this library build the whole thing for you. Don't make big changes. ZATCA will refuse the hash if ONLY ONE WHITESPACE is different that what they expect.