search

abuzuhri / Amazon-SP-API-CSharp

.Net C# library for the new Amazon Selling Partner API
MIT License
215 stars 191 forks source link

Code Hanging after amazon throws "Quota Exceeded" error #242

Closed nafberger closed 2 years ago

nafberger commented 2 years ago

the code hangs and goes into sleep when amazon throws a "Quota Exceeded" error. this is even when rate limiting is set to false, it's impossible to escape that and even putting the code in a try catch wont help. that is a problem, lets say I have 5o sellers and i wat to loop through all of them and make calls, if one fails I want to catch the error and move on, not being stuck

nafberger commented 2 years ago

@abuzuhri can u please help me on that its very important

abuzuhri commented 2 years ago

@nafberger

can set MaxThrottledRetryCount to zero or one and try again it’s in Amazonconnection

nafberger commented 2 years ago

@abuzuhri that's right. I will try but why isn't the raw headers coming back. this info is very needed in order to program intelligently against amazon's craziness.

can we please get the headers with the throttling information amazon returns. would be very helpful.

nafberger commented 2 years ago

@abuzuhri can we please get on zoom and discus that. its very important because amazon wasn't enforcing the limits till now on the reports api

abuzuhri commented 2 years ago

@abuzuhri that's right. I will try but why isn't the raw headers coming back. this info is very needed in order to program intelligently against amazon's craziness.

can we please get the headers with the throttling information amazon returns. would be very helpful.

Yes we can do that also , i can add method name (GetLastHeader)

abuzuhri commented 2 years ago

@abuzuhri can we please get on zoom and discus that. its very important because amazon wasn't enforcing the limits till now on the reports api

send me private email and we will agree on the time because we are in different time zone and its not easy

nafberger commented 2 years ago

@abuzuhri when can we talk or be on zoom, I am on new York time but I can do your time. its very important.

abuzuhri commented 2 years ago

Will try made it tomorrow my day will begin after 8 hours from now.

nafberger commented 2 years ago

@abuzuhri big thanks

tank104 commented 2 years ago

Did you manage to solve it? Anything you can share?

nafberger commented 2 years ago

@tank104 the solution is to set max retry to 0. i thanked @abuzuhri for adding the above feature to be able to see the throtling info headers from amazon

abuzuhri commented 2 years ago

Now I Add LastResponseHeader in all end point

Example 

var Headers = amazonConnection.ProductPricing.LastResponseHeader;
tank104 commented 2 years ago

Can I confirm - if I run 50 AmazonConnection in different threads (for different sellers), the rate limiting will be per seller right?

I am getting hanging issues too, but think have different setup - as I don't loop through sellers on the same AmazonConnection.

Code looks like it would be seperate, but some confirmation would be great.

mrkevinhsiao commented 2 years ago

@tank104 From my observation in production, when I run 50 AmazonConnection in different threads for different sellers, if 1 seller gets rate limited, then all threads are delayed and I couldn't figure out whether it's Amazon or the code.

50 threads * 60 seconds = 3,000 seconds delay for the next available GetOrders API call for all threads.

I had to apply a workaround to this issue, so I have some breathing space to investigate the root cause of this issue.

My current workaround is: I have 1 Azure App Service Plan scaled out to 5 instances Each instance are running 5 Web Jobs Each Web Job is running 1 Hangfire Background Server with only 1 Worker

5 Instances x 5 Web Jobs give me 25 Hangfire worker, and they are picking up Hangfire jobs to GetOrders from Amazon from different sellers

25 workers are calling GetOrders with 60secs rate limit

tank104 commented 2 years ago

I assume it must be the code - if you have it working by running in completely seperate instances right?

Does rate limiting seem fine now, and as expected at a seller level?

mrkevinhsiao commented 2 years ago

What I have noticed is that when Seller A finished GetOrders with rate limited, and then I immediately create another AmazonConnection on the same thread for Seller B to GetOrders, the call seems to be rate limited straight away without burst calls.

tank104 commented 2 years ago

@abuzuhri do you have any thoughts on this? Just wondered if you had it

abuzuhri commented 2 years ago

@tank104 its should not be like that but i will take another look on it

@tank104 and @mrkevinhsiao

I request code review also form your side maybe you found something i don't see

https://github.com/abuzuhri/Amazon-SP-API-CSharp/blob/569a4a16477bd8e35b12e0d0aff1a88dd1ab54a4/Source/FikaAmazonAPI/Services/RequestService.cs#L172

tank104 commented 2 years ago

I already checked and it looked fine. No static or singleton that’s I could see. So expect a complete new Amazon connection for each seller.

I have logged a ticket with Amazon.

@abuzuhri donyoubuse with multi sellers? If so do you have rate issues across sellers?

@mrkevinhsiao do you use it in pure sp-api mode or are you using converted MWS to sp-api tokens? I am using all MWS converted tokens

mrkevinhsiao commented 2 years ago

@tank104 I converted roughly 1,000 sellers from MWS tokens to SP-API tokens.

I do have several sellers that recently authorized my Amazon app without MWS token conversion, so I will check the logs on my side and see how long did the GetOrders calls take for them.

tank104 commented 2 years ago

I am not familiar with RestSharp - so not sure if the issue lies there. One thing I do know - for .NET framwork DEfault parallel connections to a specific domain is set to 2. And needs to be changed to this: ServicePointManager.DefaultConnectionLimit = 100;

I believe .net core wisely has this set to int32.maxvalue.

This I found: https://stackoverflow.com/questions/55857588/why-is-restsharp-limiting-concurrent-connections-to-the-same-server-to-2

However we are already setting it to 100 and get the issue.

tank104 commented 2 years ago

Ok I have a test case setup for this so can replicate easily (can't share as loaded with credentials.

Looking at request/response it doesn't return 409, yet the library throttles it.
So should have a solution shortly..

tank104 commented 2 years ago

This is the culprit: internal Dictionary<RateLimitType, RateLimits> UsagePlansTimings { get; set; } = RateLimitsDefinitions.RateLimitsTime;

You are seeting the same rate limits for all AmazonConnections - and there is RequestsSent counter in that so they all increment the same. Will get PR up shortly.

tank104 commented 2 years ago

That was fun to investigate :) @abuzuhri can we get this in and package built ASAP? Fortunately MWS hasn't been switched off yet, but was supposed to be Monday, so any day now.

https://github.com/abuzuhri/Amazon-SP-API-CSharp/pull/270

abuzuhri commented 2 years ago

@tank104

Please check v1.4.14

https://www.nuget.org/packages/CSharpAmazonSpAPI/