Improving use under heavy loads

[resistorsoftware]

I am using this library to create 1000 products. It often blows out on either the product create or publish call.

A sample error stream would be on the request itself, with this:

SignalException: SIGTERM
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:229:in `wait_readable'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:229:in `rbuf_fill'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:199:in `readuntil'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:209:in `readline'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:158:in `read_status_line'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:147:in `read_new'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1862:in `block in transport_request'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1853:in `catch'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1853:in `transport_request'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1826:in `request'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1819:in `block in request'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1238:in `start'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1817:in `request'
/app/vendor/bundle/ruby/3.2.0/gems/httparty-0.21.0/lib/httparty/request.rb:156:in `perform'
/app/vendor/bundle/ruby/3.2.0/gems/httparty-0.21.0/lib/httparty.rb:612:in `perform_request'
/app/vendor/bundle/ruby/3.2.0/gems/httparty-0.21.0/lib/httparty.rb:542:in `post'
/app/vendor/bundle/ruby/3.2.0/gems/httparty-0.21.0/lib/httparty.rb:649:in `post'
/app/vendor/bundle/ruby/3.2.0/gems/shopify_api-12.5.0/lib/shopify_api/clients/http_client.rb:43:in `request'
/app/vendor/bundle/ruby/3.2.0/gems/sorbet-runtime-0.5.10827/lib/types/private/methods/call_validation_2_7.rb:106:in `bind_call'
/app/vendor/bundle/ruby/3.2.0/gems/sorbet-runtime-0.5.10827/lib/types/private/methods/call_validation_2_7.rb:106:in `block in create_validator_method_fast1'
/app/vendor/bundle/ruby/3.2.0/gems/shopify_api-12.5.0/lib/shopify_api/clients/graphql/client.rb:35:in `query'
/app/vendor/bundle/ruby/3.2.0/gems/sorbet-runtime-0.5.10827/lib/types/private/methods/call_validation.rb:153:in `bind_call'
/app/vendor/bundle/ruby/3.2.0/gems/sorbet-runtime-0.5.10827/lib/types/private/methods/call_validation.rb:153:in `validate_call_skip_block_type'
/app/vendor/bundle/ruby/3.2.0/gems/sorbet-runtime-0.5.10827/lib/types/private/methods/call_validation.rb:95:in `block in create_validator_slow_skip_block_type'
/app/vendor/bundle/ruby/3.2.0/gems/shopify_graphql-1.1.1/lib/shopify_graphql/client.rb:8:in `execute'
/app/vendor/bundle/ruby/3.2.0/gems/shopify_graphql-1.1.1/lib/shopify_graphql/mutation.rb:10:in `execute'

So I would like to have this call simply wait say 5 seconds, then retry itself. I am interested in how to best code that up.

My sample calling code is straight out of the docs and blows out on line one of the call method.

class CreateProduct
  include ShopifyGraphql::Mutation

  MUTATION = <<~GRAPHQL
    mutation($input: ProductInput!, $media: [CreateMediaInput!]) {
      productCreate(input: $input, media: $media) {
        product {
          handle
          id
        }
        userErrors {
          field
          message
        }
      }
    }
  GRAPHQL

  def call(input:, media:)
    response = execute(MUTATION, input: input, media: media)
    response.data = response.data.productCreate
    handle_user_errors(response.data)
    response
  rescue => e
    puts "Error in CreateProduct: #{e.message}, #{response}"
  end
end

[kirillplatonov]

When making thousands of API calls you inevitably will hit Shopify 5XX or connection errors. You could handle it using Ruby rescue, sleep, and retry. Or extract calls to ActiveJob so it will retry it for you.

[resistorsoftware]

My point was, having to babysit the GQL by writing in a rescue retry for every single API call is perhaps the worst of the good options, and maybe this could be built-in to this module as some kind of configurable thing. If you don't want it here, then monkey-patching and unique individual customization is 100% possible, but less than desirable for general use!

[resistorsoftware]

SignalException ---> SignalException: SIGTERM

So this is tricky! Ruby deals with this via HTTParty which then extends to ShopifyAPI before finally dying on the hill of my task and Shopify GraphQL and rake.

Even wrapping a SignalException rescue does not save the day. Straight up, due to some kind of serious bad news bears, the concept of looping and doing API calls dies badly. Yay!