Random ETIMEDOUT when making lots of graphql requests

[tommypepsi]

Issue summary

I'm updating and creating lots of products with each of them having multiple request (creating the products, adding metafields and adding international price for each markets). I made a queue that handles respecting the available graphql points so I'm not hitting the throttle response by shopify. Somehow I seem to be randomly getting a timedout error after a couple hundreds of request.

At first I thought I was too greedy with my concurrent requests but setting the concurrent to 10, 5 or 2 all get the same result. It only seems to not happen when I only do one request at a time but this is definitely too slow.

I also tried adding "tries: 10" to my request but that doesn't seem to work, it still throws an error without retrying.

The only way I managed to make it work is by doing a try catch and retrying the request manually, then it seems to work the second time.

Expected behavior

Not sure if the error is an issue related to this library, if so I guess the expected behavior is to not get that error. Otherwise I would expect the tries option to actually retry the request but it doesn't seem to be the case.

Actual behavior

I get this error after a couple of request:

HttpRequestError: Failed to make Shopify HTTP request: FetchError: request to https://mystore.myshopify.com/admin/api/2022-04/graphql.json failed, reason: connect ETIMEDOUT 23.227.38.74:443
     at HttpRequestError.ShopifyError [as constructor] (C:\Users\.....\Documents\Code\Shopify\Apps\.....\node_modules\@shopify\shopify-api\dist\error.js:13:28)
     at new HttpRequestError (C:\Users\.....\Documents\Code\Shopify\Apps\.....\node_modules\@shopify\shopify-api\dist\error.js:79:42)
     at C:\Users\.....\Documents\Code\Shopify\Apps\.....\node_modules\@shopify\shopify-api\dist\clients\http_client\http_client.js:261:35
     at runMicrotasks (<anonymous>)
     at processTicksAndRejections (node:internal/process/task_queues:96:5)

I tried to add the tries option like that:

await client.query({
  data: {
    query: mutationString,
    variables: {
      input: productData,
    },
    tries: 10,
  },
});

//and like that
await client.query({
  data: {
    query: mutationString,
    variables: {
      input: productData,
    },  
  },
  tries: 10,
});

The only way I managed to make it work is by doing something like that:

try{
  await client.query({
    data: {
      query: mutationString,
      variables: {
        input: productData,
      },
    },
  });
}
catch(err){
  retryRequest();
}

In the retry in the catch so far it always seems to be working. So I'd expect the tries option to do the same thing instead of having to do a catch and retry manually.

[uschtwill]

We've been seeing the same thing, but for REST calls against the Admin API, same IP too.

{"responseMessage":"connect ETIMEDOUT 23.227.38.74:443"}

A lot more than what you are describing though, see: https://community.shopify.com/c/shopify-apis-and-sdks/large-number-of-timeouts/td-p/1666574

Wondering if it's related. Shopify support is not helpful.

[mkevinosullivan]

@tommypepsi The retriable nature of the query() call relates to receiving particular response errors from the Shopify system and re-trying on the applications behalf, e.g., 429 or 500 errors. ETIMEDOUT errors, similar to, say, ECONNREFUSED errors, are lower-level TCP/IP errors that the library immediately surfaces to the application, by design.

[mkevinosullivan]

@uschtwill Based on the message left in the community forum, it appears you are using https://github.com/MONEI/Shopify-api-node, which is a 3rd party library, not supported by Shopify.

[StefanoDiLegami]

I have the same issue. I am using Shopify API Library and google PubSub to manage a lot of Webhooks, but when I have to many concurrent request to the Admin API, they fail with ETIMEDOUT or ECONNRESET but in the Shopify docs there is nothing mentioned about a request limit for GraphQL, only calculated query cost and I am not using more than allowed because I implemented a throttling method for the query cost.

[hrstrand]

I have the same issue. I am using Shopify API Library and google PubSub to manage a lot of Webhooks, but when I have to many concurrent request to the Admin API, they fail with ETIMEDOUT or ECONNRESET but in the Shopify docs there is nothing mentioned about a request limit for GraphQL, only calculated query cost and I am not using more than allowed because I implemented a throttling method for the query cost.

@StefanoDiLegami did you solve this issue? I'm running into the exact same scenario, where I have to react on incoming events from Shopify and is hit with these timeouts if there are many events in a short period of time. Some requests take up to 60 seconds to complete, others time out, and some complete as expected

[StefanoDiLegami]

I have the same issue. I am using Shopify API Library and google PubSub to manage a lot of Webhooks, but when I have to many concurrent request to the Admin API, they fail with ETIMEDOUT or ECONNRESET but in the Shopify docs there is nothing mentioned about a request limit for GraphQL, only calculated query cost and I am not using more than allowed because I implemented a throttling method for the query cost.

@StefanoDiLegami did you solve this issue? I'm running into the exact same scenario, where I have to react on incoming events from Shopify and is hit with these timeouts if there are many events in a short period of time. Some requests take up to 60 seconds to complete, others time out, and some complete as expected

Unfortunately not. Just trying to repeat the request to Shopify API as long as it needs to complete successfully.

[hrstrand]

I solved my issue in the regards - it turned out to be a low limit in my GCE Cloud-NAT settings. ( below is from my reply to same issue in the Shopify forums : https://community.shopify.com/c/shopify-apis-and-sdks/large-number-of-timeouts/m-p/1995314/highlight/true#M89101 )

By spinning up instances of GCE vm's, I found that only the ones using a Cloud NAT had the issues. VM's with an external IP did not have the issue.

So the fix was actually very easy - I am no expert in Cloud NAT, but I found a setting that indicated a maximum of 64 connections to a given ip:port destination per vm. I bumbed that to 4096, and now everything works with no slow connections or timeout.

I tested it with this from a linux command line

siege -t 120s -c 10 -v https://shapingnewtomorrow.myshopify.com/collections.json

I got around 4500 transactions in 2 minutes

I hope you can apply similar changes to your setup and get it working

thanks,

Peter