Seldom getting InvalidWebhookError: Could not validate request for topic

[mariusa]

Issue summary

Webhooks work fine, but now I just imported 3k products and got 11 validation errors

InvalidWebhookError: Could not validate request for topic products/create

without any detail.

Expected behavior

Webhook validation should always work, since these are webhooks received from Shopify.

1st step would be to show some useful information when validation fails, I will make a PR. Then I should be able to reproduce this, maybe it happens on specific products, and find out more.

Actual behavior

InvalidWebhookError: Could not validate request for topic products/create

Steps to reproduce the problem

Need to get some debugging info in order to reproduce. Will make a PR.

---

Checklist

• [x] I have described this issue in a way that is actionable (if possible)

[mariusa]

Managed to reproduced this: import the attached set of products (1st fail, the rest 2 work). BUT, it fails only on one store:

X-Shopify-Topic
products/create
X-Shopify-Hmac-Sha256
2Zuy/weLo3ChSlVbu/UXkEOHslVC9dWd1PCic8tkpYw=
X-Shopify-Shop-Domain
test-eus.myshopify.com
X-Shopify-API-Version
2022-01
X-Shopify-Webhook-Id
6b699da2-2661-4baf-9443-eb770f802732

Deleting products, importing again fails again. hmac generated by nodejs is generatedHashRaw:

{"X-Shopify-Topic":"products/create","generatedHashRaw":"dmjeA3Y14ueXhLC7khc7fWdeqTt9HSPuM83ExFDiccw=","X-Shopify-Product-Id":"6698552164425","X-Shopify-Hmac-Sha256":"s/lSw1JBBBYsPmk5/99PqUIjivW0opK7hIDyALj1E3g=","X-Shopify-Shop-Domain":"test-eus.myshopify.com"}

X-Shopify-Webhook-Id
11cde2f9-155a-44ba-a617-ca2001ac5fff

The same import works fine on another store, with the same app and webook handling code. The code to generate HMAC in nodejs gives a different value than Shopify HMAC header, just for this product and store. https://github.com/Shopify/shopify-node-api/blob/62abcf175fcb5acf8eaa38b93868d3bb443228e4/src/webhooks/registry.ts#L383

Probably related to #275

[mariusa]

shopify-products-import-hmac-error.csv

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will be closed if no further action occurs in 14 days.

[mariusa]

not stale

[oribenez]

How can i fix this issue for now?

[pacocastrotech]

Hello 👋

I've been investigating recently some of the open issues related with HMAC validation.

I started with this one and, while I did not import the products, I emulated the products/create webhooks that'd been generated from the csv file (shopify-products-import-hmac-error.csv).

I was able to receive and validate them for all the cases. I used my app key and the basic app template you get when you create an app using the last version of the CLI (I just added the webhook handler).

I also did the validation twice (before it left the server, and in the client app) to try to spot where the problem could be. I suspected an encoding issue or possibly a wrong secret being used, but it all worked fine for me for all rows.

Can anyone confirm this is still happening, for which version of the library, and if it happens too with the most recent one?

[mariusa]

Hello,

Thanks for looking into this. It still happens with 5.2.0, node 18 (just upgraded now from node 16)

It happens only for some products, on some shops. I tried the same product exported & imported in another shop, that worked. It could be an encoding issue, which gets changed/fixed during export / import. Now I can't replicate with the previous exported products above.

Could I send you privately by email a list of failed validations on production shops that happened in the past hour? It happened on orders/create and products/update (X-Shopify-Order-Id, X-Shopify-Product-Id, X-Shopify-Shop-Domain, X-Shopify-Hmac-Sha256)

[mariusa]

I've edited registry.js to console.log(req.body) when hmac != generatedHash

Examples when they are different -- notice the m��necă chars, it might be related. These show fine in Shopify admin: mânecă. I assume that this product will be saved from Shopify admin, the encoding is fixed and won't happen again (I tried this with another product).

Product id: "gid://shopify/Product/6605095501888"

{"id":6605095501888,"title":"Body seamless din lână merinos și mătase - Natur","body_html":"\u003cp\u003e\u003cspan\u003eBody seamless cu mânecă scurtă din lână merinos organică și mătase.\u003c\/span\u003e\u003c\/p\u003e\n\u003cul\u003e\n\u003cli\u003efără cusături laterale, tricotat tubular\u003c\/li\u003e\n\u003cli\u003e\u003cspan\u003edecolteu cu croială plic\u003c\/span\u003e\u003c\/li\u003e\n\u003cli\u003e\u003cspan\u003ese închide cu capse în zona scutecului\u003c\/span\u003e\u003c\/li\u003e\n\u003cli\u003e\u003cspan\u003etricot elastic\u003c\/span\u003e\u003c\/li\u003e\n\u003cli\u003e\u003cspan\u003ese poate purta ca prim strat în zilele foarte reci și ca atare în zilele calde\u003c\/span\u003e\u003c\/li\u003e\n\u003cli\u003emătasea naturală din compoziție este hipoalergenică și este recomandată pentru cei cu pielea sensibilă\u003cbr\u003e\n\u003c\/li\u003e\n\u003c\/ul\u003e\n\u003cp\u003e\u003cspan\u003eMateriale și sustenabilitate\u003c\/span\u003e\u003c\/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cspan\u003e70% lână merinos organică, 30% mătase mulberry, jersey plain knit\u003c\/span\u003e\u003c\/li\u003e\n\u003cli\u003e\u003cspan\u003efinețea fibrei de lână: 19.5 micron (extrafine)\u003c\/span\u003e\u003c\/li\u003e\n\u003cli\u003e\u003cspan\u003ecertificări: Best și GOTS\u003c\/span\u003e\u003c\/li\u003e\n\u003cli\u003e\u003cspan\u003eprocesul de producție respectă cele mai stricte standarde ecologice ale organizației IVN (International Association of Natural Textile Industry)\u003c\/span\u003e\u003c\/li\u003e\n\u003cli\u003e\u003cspan\u003ese recomandă spălarea de mână, în apă rece\u003c\/span\u003e\u003c\/li\u003e\n\u003cli\u003e\u003cspan\u003edesign Cosilana - Germania; fabricat în Germania.\u003c\/span\u003e\u003c\/li\u003e\n\u003c\/ul\u003e\n\u003cul\u003e\u003c\/ul\u003e\n\u003cul\u003e\u003c\/ul\u003e\n\u003cul\u003e\u003c\/ul\u003e","vendor":"Cosilana","product_type":"Haine","created_at":"2022-06-05T12:16:23-04:00","handle":"body-seamless-din-lana-merinos-si-matase-natur","updated_at":"2022-12-02T16:12:29-05:00","published_at":"2022-06-05T12:16:21-04:00","template_suffix":null,"status":"active","published_scope":"web","tags":"baby, basic, blackfriday-2021, body-maneca-scurta, Color-Natur, continue-aw2020, continue-ss2020, cosilana, Cosilana-Q71, general-care, haine, iarna, lana de vara, layer-first, layer-mid, marime-haine-104, marime-haine-44-56, marime-haine-50, marime-haine-50-56, marime-haine-56-62, marime-haine-62-68, marime-haine-68, marime-haine-68-74, marime-haine-74-80, marime-haine-80, marime-haine-86, marime-haine-86-92, marime-haine-92, marime-haine-98, marime-haine-98-104, masterSKU-cos-71052, material-lana, material-lana-si-matase, micron-extrafine, oferta-first-layer, off-info-continue-aw2019, on-sale, onSale, promo, superdeals-2021, toamna, vara","admin_graphql_api_id":"gid:\/\/shopify\/Product\/6605095501888","metafields":[],"private_metafields":[],"variants":[{"id":39436647989312,"product_id":6605095501888,"title":"50\/56","price":"38.80","sku":"cos-71052-01-50","position":1,"inventory_policy":"deny","compare_at_price":"97.00","fulfillment_service":"manual","inventory_management":"shopify","option1":"50\/56","option2":null,"option3":null,"created_at":"2022-06-05T12:16:24-04:00","updated_at":"2022-12-02T16:10:10-05:00","taxable":true,"barcode":"4053175002689","grams":0,"image_id":null,"weight":0.0,"weight_unit":"kg","inventory_item_id":41530904576064,"inventory_quantity":0,"old_inventory_quantity":0,"requires_shipping":true,"admin_graphql_api_id":"gid:\/\/shopify\/ProductVariant\/39436647989312","metafields":[]},{"id":39436648022080,"product_id":6605095501888,"title":"62\/68","price":"30.90","sku":"cos-71052-01-62","position":2,"inventory_policy":"deny","compare_at_price":"103.00","fulfillment_service":"manual","inventory_management":"shopify","option1":"62\/68","option2":null,"option3":null,"created_at":"2022-06-05T12:16:24-04:00","updated_at":"2022-12-01T11:00:01-05:00","taxable":true,"barcode":"4053175002696","grams":0,"image_id":null,"weight":0.0,"weight_unit":"kg","inventory_item_id":41530904608832,"inventory_quantity":0,"old_inventory_quantity":0,"requires_shipping":true,"admin_graphql_api_id":"gid:\/\/shopify\/ProductVariant\/39436648022080","metafields":[]},{"id":39436648054848,"product_id":6605095501888,"title":"74\/80","price":"43.60","sku":"cos-71052-01-74","position":3,"inventory_policy":"deny","compare_at_price":"109.00","fulfillment_service":"manual","inventory_management":"shopify","option1":"74\/80","option2":null,"option3":null,"created_at":"2022-06-05T12:16:24-04:00","updated_at":"2022-12-02T16:08:59-05:00","taxable":true,"barcode":"4053175002702","grams":0,"image_id":null,"weight":0.0,"weight_unit":"kg","inventory_item_id":41530904641600,"inventory_quantity":0,"old_inventory_quantity":0,"requires_shipping":true,"admin_graphql_api_id":"gid:\/\/shopify\/ProductVariant\/39436648054848","metafields":[]},{"id":39436648087616,"product_id":6605095501888,"title":"86\/92","price":"46.00","sku":"cos-71052-01-86","position":4,"inventory_policy":"deny","compare_at_price":"115.00","fulfillment_service":"manual","inventory_management":"shopify","option1":"86\/92","option2":null,"option3":null,"created_at":"2022-06-05T12:16:24-04:00","updated_at":"2022-12-02T16:12:29-05:00","taxable":true,"barcode":"4053175002719","grams":0,"image_id":null,"weight":0.0,"weight_unit":"kg","inventory_item_id":41530904674368,"inventory_quantity":0,"old_inventory_quantity":0,"requires_shipping":true,"admin_graphql_api_id":"gid:\/\/shopify\/ProductVariant\/39436648087616","metafields":[]},{"id":39436648120384,"product_id":6605095501888,"title":"98\/104","price":"47.60","sku":"cos-71052-01-98","position":5,"inventory_policy":"deny","compare_at_price":"119.00","fulfillment_service":"manual","inventory_management":"shopify","option1":"98\/104","option2":null,"option3":null,"created_at":"2022-06-05T12:16:24-04:00","updated_at":"2022-12-02T16:11:33-05:00","taxable":true,"barcode":"4053175002726","grams":0,"image_id":null,"weight":0.0,"weight_unit":"kg","inventory_item_id":41530904707136,"inventory_quantity":0,"old_inventory_quantity":0,"requires_shipping":true,"admin_graphql_api_id":"gid:\/\/shopify\/ProductVariant\/39436648120384","metafields":[]}],"options":[{"id":8488752840768,"product_id":6605095501888,"name":"Marime haine","position":1,"values":["50\/56","62\/68","74\/80","86\/92","98\/104"]}],"images":[{"id":28440266145856,"product_id":6605095501888,"position":1,"created_at":"2022-06-05T12:16:24-04:00","updated_at":"2022-06-05T12:16:24-04:00","alt":"Body seamless din lână merinos și mătase - Natur-Cosilana-HipHip.ro","width":1434,"height":2048,"src":"https:\/\/cdn.shopify.com\/s\/files\/1\/0079\/1910\/8160\/products\/body-cosilana-cu-maneca-scurta-lana-si-merinos-matase-natur-cosilana-hiphipro_2.jpg?v=1654445784","variant_ids":[],"admin_graphql_api_id":"gid:\/\/shopify\/ProductImage\/28440266145856","metafields":[]},{"id":28440266178624,"product_id":6605095501888,"position":2,"created_at":"2022-06-05T12:16:24-04:00","updated_at":"2022-06-05T12:16:24-04:00","alt":"Body Cosilana cu m��necă scurtă lână merinos si mătase - Natur-Cosilana-HipHip.ro","width":1434,"height":2048,"src":"https:\/\/cdn.shopify.com\/s\/files\/1\/0079\/1910\/8160\/products\/body-cosilana-cu-maneca-scurta-lana-merinos-si-matase-natur-cosilana-hiphipro_a0e33066-1e68-4d3d-b739-53d1535c260c.jpg?v=1654445784","variant_ids":[],"admin_graphql_api_id":"gid:\/\/shopify\/ProductImage\/28440266178624","metafields":[]}],"image":{"id":28440266145856,"product_id":6605095501888,"position":1,"created_at":"2022-06-05T12:16:24-04:00","updated_at":"2022-06-05T12:16:24-04:00","alt":"Body seamless din lână merinos și mătase - Natur-Cosilana-HipHip.ro","width":1434,"height":2048,"src":"https:\/\/cdn.shopify.com\/s\/files\/1\/0079\/1910\/8160\/products\/body-cosilana-cu-maneca-scurta-lana-si-merinos-matase-natur-cosilana-hiphipro_2.jpg?v=1654445784","variant_ids":[],"admin_graphql_api_id":"gid:\/\/shopify\/ProductImage\/28440266145856","metafields":[]}}

[pacocastrotech]

That was a good clue, thanks! I've checked the alt field and all is ok. Indeed one of the images alt is fine and the other is not. I still don't see a clear cause. So far it looks like the encoding is fine in the admin but when you retrieve it via the GraphQL api or in the webhook payload it's wrong? 🤔
Could you please send me the X-Shopify-Webhook-Id (just that field would be enough for me) for some recent failures?

[mariusa]

Sure, here are some from past hour: products/update

X-Shopify-Webhook-Id 21789976-40e6-41a2-bdf9-0eabf533fd05
X-Shopify-Webhook-Id fc1955df-7960-4bde-8345-8c0cd69bff9f
X-Shopify-Webhook-Id 1f7fbd0f-bd7d-4a36-8bb0-ab06528d06f4

X-Shopify-Webhook-Id b0c23a05-876b-440f-a29f-c6959ee95637

X-Shopify-Webhook-Id 1e2890fd-4dcf-4d2b-8119-4140a991a9e0

for topic orders/create, on different shops:

X-Shopify-Webhook-Id 2488732e-8694-4c24-82ff-0e26037c2e17
X-Shopify-Webhook-Id b4001009-0212-4e0b-997b-84daf969ee50
X-Shopify-Webhook-Id 7a57cfb8-a369-40f5-8eff-82bde5187e59
X-Shopify-Webhook-Id 0ddca721-7328-4a5e-b717-69e722f5dd5f

[pacocastrotech]

Hello 👋 , I checked all these cases one by one and the pattern seems to hold: the first delivery fails, but our retries mechanism starts and they are finally delivered. Some of them were delivered a minute later, some took longer but they all made it to the destination url and they were accepted.

I can't see the payload but I see that for each retry, the hmac field changes. Reason for this: retries are carried out during 48h and we re-read the entity info each time to return the most current payload possible. Usually this happens due to changes in state or time/date values. With such short time-span I'd doubt that there was a significant change in descriptions or text fields.

Are those delays causing troubles to the app? Here there are some best practices we recommend: https://shopify.dev/apps/webhooks/best-practices

I'll investigate the ratio of retries in those shops and the ratio of max retries performed in case it was high but, I'd say that if there was an issue with the hmac code, all the retries should be failing the 18 times we send them. And for these cases, the worst case I've seen it was 7. Most of them were delivered at the second attempt or the third, which happens in a matter of minutes.

[mariusa]

Hello! Thanks for looking into this.

Are those delays causing troubles to the app? Here there are some best practices we recommend

I don't understand this part of the reply :) We call the Shopify code which does the validation, which should then should call our processing handlers. It doesn't get to processing because the the validation fails with a very specific error: 401 InvalidWebhookError: Could not validate request for topic products/update

It was was problem with our servers, it would timeout or return 5xx, right? If it was any error in our handlers, I see it's a different statusCode:

try {
            await handler.callback(
              graphqlTopic,
              domain,
              rawBody,
              webhookId,
              apiVersion,
            );
            response.statusCode = StatusCode.Ok;
          } catch (error) {
            response.statusCode = StatusCode.InternalServerError;
            errorMessage = error.message;
          }

I can't see the payload but I see that for each retry, the hmac field changes. Reason for this: retries are carried out during 48h and we re-read the entity info each time to return the most current payload possible. Usually this happens due to changes in state or time/date values.

So the product info does change in the shop? There are no retries with the same hmac which succeed later? This does point to an issue in hmac computation.

Some other facts:

1. it only happens for products/update and orders/create
2. it happens a lot on same shops, and very seldom on others

We'll add payload to logging for a short time (it will clug the logs), hoping to get more clues.

[mariusa]

I looked at some Webhook-Ids: 8b9815c1-b61d-43b2-960d-f3620d9468ae b7a4fcda-fb61-4b8a-b5a9-9de475686464

It always had the same HMAC, and was accepted the 8th and 7th time. All the failures were 401 Could not validate request. But why the same request worked later? So strange.

[pacocastrotech]

I don't understand this part of the reply :) We call the Shopify code which does the validation, which should then should call our processing handlers. It doesn't get to processing because the the validation fails with a very specific error: 401 InvalidWebhookError: Could not validate request for topic products/update

Hello, I was trying to assess the impact a little bit better, because you end up receiving those webhooks (after retries), I was wondering if the delay causes problems in your app.

So the product info does change in the shop? There are no retries with the same hmac which succeed later? This does point to an issue in hmac computation.

Here's an interesting case:

Webhook-Id b0c23a05-876b-440f-a29f-c6959ee95637 required 8 attempts.

The 3 first ones (at 7:49, 7:50, and 7:52) arrived with, say, HMAC_A. The 5 following ones (at 7:54, 7:59, 8:21, 8:51 and 9:51) arrive with HMAC_B

from those 5 last ones, the first 4 ones fail and the last one is accepted. Can you confirm that for all those failures what you saw was a Could not validate request error?

It was was problem with our servers, it would timeout or return 5xx, right?

Not necessarily: The error says that we don't get the same HMAC when we encode the payload with the client secret key.

1. If you were using a different key in your servers (occasionally) than the one we have for you in ours, those validations would fail.
2. If the encoding you receive is not the same that we use in our servers, the validation would fail
3. If there's an issue with the library (either setting the encoding of the response or applying the validation), the validation would fail

I'm not discarding 2 or 3 at all. I'm just gathering information to find out if it's any of those cases and to ask other teams about the case. Is your App build using the ShopifyCLI? Or you built your own custom integration using the API libraries?

[mariusa]

Here's an interesting case:

That's from Dec 5, but we can see logs only from past 2 days :( In Shopify they all show with 401, so yes, they were all invalid. Also in Shopify we do have some few other failures (Timeout, 503), so logging & statuses work as they should, it's not always 401. I think it just happened that the shop updated the product data meanwhile.

I did send 2 other recent examples above with the same HMAC, which finally got accepted. This ruins my theory that the library validation code stumbles on some specific encoding. There are still some cases where it retries up to 16 times and still 401: 2ef973e5-8182-4970-bc3a-037d01abdb0e

Thanks for listing the 3 possible examples when this would happen.

We built your own custom integration using this library. We install & use it from npm, we don't have a local copy that we modify or something.

This weekend we will update to v6, curious if that brings a change.

[mariusa]

Guess what? After updating to v6, no more validation errors. I guess it's solved by having webhook processing receiving rawBody from the app rather than parsing it by itself, and maybe not handling some encodings/cases well. Another reason to have it just receive data from the app (body, headers) and returning data to app, rather than taking control and assuming Express is used everywhere https://github.com/Shopify/shopify-api-js/issues/624

Thanks for your help, closing this now.

[TheSecurityDev]

I'm getting this issue and still on v5 using Koa. Is there any workaround without having to upgrade?

[goranhorvathr]

We did an express app following the documentation by the book https://github.com/Shopify/shopify-api-js/blob/main/packages/shopify-api/docs/guides/custom-store-app.md and etc. Raised several support tickets. Built 3 apps remix with cli, Vue custom & express one following the documentation & creating a private app / custom app in the Shopify itself. Neither worked and HMAC always said "Could not validate request HMAC".

Debugged for countless hours but the ORDERS_CREATE Webhook just does not work for us (mad sure the urls are correct, all keys are correct). Everything else works just fine we can add gift cards programmatically which we needed we can query admin api without a problem but we cannot subscribe to order success event which is the most important thing we need so we can programmatically create and send gift cards to the customers on Cyber Monday.

I am truly and utterly beaten short deadline + Shopify code and documentation is a massive mess. In my career of 20 years of development this is the first time that there was something that was unsolvable as this. Tomorrow ill try last thing and thats Python script to check validity of the Webhook since its not automated and using shopify-api which is bugged and if it works ill perhaps be able to deliver the application for the client.

Documentation is outdated, contradictory one says put in hostname the ngrok url on other page it says for shopifyApi put the myshopify url of the website, private apps on the stores which are now custom apps do not need oAuth yet registering webhooks requires oAuth so even if its not failing we don't know if its working. write_gift_cards, read_gift_cards permissions have to be given by Shopify support which is mind-blowing when we create apps with CLI and add the scopes scopes should just work. No necessary input from anyone. When user client installs the app or it gets deployed on development store it should just work.

And this is not the first time something that should be working seamlessly does not work or we end up disappointing our clients and in the long run burn our hours because we think things will work.

The main issue is it works so sporadically its ridiculous. Its working and not working at the same time. Shopify team mastered the Schrodinger's theory. You guys are in superposition.