Excessive requests on v1.oauth2.token API

[westhouseit]

Description

The PayPal v1.oauth2.token API is being called too many times thus causing RATE_LIMIT_REACHED on the checkout API which breaks checkout.

Steps to reproduce

I can't reproduce exactly since PayPal won't give full details of everything involved in this issue and it appears Craft Commerce doesn't store anything about the auth requests.

Additional info

• Craft version: Pro 4.2.1.1
• Commerce PayPal Checkout version: 2.1.0.2
• Commerce version: 4.1.0
• PHP version: 8.1.9
• Database driver & version: MySQL 10.2.33
• Other Plugins & versions: Bulk Edit 4.0.0-RC1 Contact Form 3.0.0 Craft Commerce 4.1.0 Environment Label 4.0.2 Feed Me 5.0.4 Formie 2.0.5 Form Management 4.0.0 Guest Entries 3.0.1 Redactor 3.0.2

Additional context

From PayPal:

Unfortunately, I cannot provide too many details about the specific reason that cause this "rate limit" error, because the specific rule of Rate Limit policy is part of our information security which is not public info.

I can only tell you that I see the errors were almost occurring in the "v1.oauth2.token" API, which is used to get the access-token for calling other APIs. You can find more details in this document: https://developer.paypal.com/api/rest/authentication/

Please lower the frequency of calling this API, and I'd like to point out that PayPal returns an access-token and the number of seconds the access-token is valid, which means you can reuse the access token until it expires, and before that, you don't need to call this API again. That might help you to lower the frequency of calling this API significantly.

[westhouseit]

It looks like this other error is happening due to the rate limiting. On the front-end users are seeing:

Error: Expected an order id to be passed.

[westhouseit]

Is there a better place to post this?

[samueldraper]

@nfourtythree we have three craft commerce sites all experiencing the same issues of "expected order id to be passed" and "RATE_LIMIT_REACHED" errors. Two of them have been upgraded to Craft4 and Commerce4 hoping that would resolve but problem persists.

Is there a way to escalate this as requiring urgent attention?

[westhouseit]

index.php?p=actions/commerce/payments/pay

[samueldraper]

Providing some extra info in case it helps narrow down how/when the problem was introduced.

Client A (currently on Craft 4) 02/08/22 payment error issues reported 18/07/22 site updated to Craft 3.7.48 25/05/22 site updated to Craft 3.7.43 and Commerce 3.4.15

Client B (currently on Craft 4) 27/07/22 payment error issues reported 18/07/22 site updated to Craft 3.7.48 11/07/22 site updated to Craft 3.7.47.1 and Commerce 3.4.15

Client C (currently still on Craft 3) 08/08/22 payment error issues reported 18/07/22 site updated to Craft 3.7.48 26/05/22 site updated to Craft 3.7.43 and Commerce 3.4.15

[angrybrad]

Error: Expected an order id to be passed.

When that error happens, can you get a full stack trace leading up to it in Craft's logs? If so, can someone share that log file with support@craftcms.com and reference this ticket?

[westhouseit]

I'll see what I can get. In the meantime, I was able to replicate by clicking the Debit or Credit Card button and closing repeatedly. The 19th request to index.php?p=actions/commerce/payments/pay returned the error.

Are API auth requests being cached in Commerce? Otherwise multiple checkout attempts from the server IP looks to be the issue.

[samueldraper]

It looks like this other error is happening due to the rate limiting. On the front-end users are seeing:

Error: Expected an order id to be passed.

Sometimes the customer sees the above error in a browser warning dialog.

Other times there is a rate limit reached error shown in the website's flash message (i.e. from P&T Commerce example templates).

And the payment failures show in the Commerce CP as: Gateway Message {"name":"RATE_LIMIT_REACHED","debug_id":"[redacted]","message":"Too many requests. Blocked due to rate limiting.","links":[]}

[nfourtythree]

Hi All

Thank you for bringing this to our attention.

We wanted to let you know that we are looking into a solution for this. We believe PayPal may have tweaked its rate limiting threshold, which is why we have seen an increase.

Also, the issue appears to be in the PayPal Checkout SDK which is the underlying library this plugin uses in its communication with the PayPal REST API. Unfortunately, this library from PayPal now appears to have been deprecated in July. This forces us to figure out the best method to fix the issue while causing the least disruption to developers.

This issue will be updated with any progress, we hope to have a solution out as soon as possible.

Thanks,

[samueldraper]

Hi All

Thank you for bringing this to our attention.

We wanted to let you know that we are looking into a solution for this. We believe PayPal may have tweaked its rate limiting threshold, which is why we have seen an increase.

Also, the issue appears to be in the PayPal Checkout SDK which is the underlying library this plugin uses in its communication with the PayPal REST API. Unfortunately, this library from PayPal now appears to have been deprecated in July. This forces us to figure out the best method to fix the issue while causing the least disruption to developers.

This issue will be updated with any progress, we hope to have a solution out as soon as possible.

Thanks,

Thank you. Are you able to provide a rough ETA (hours, days, weeks) so that we can help our clients decide a course of action to accept payments in the interim.

[westhouseit]

Also, the issue appears to be in the PayPal Checkout SDK which is the underlying library this plugin uses in its communication with the PayPal REST API.

Thanks for looking into this. As mentioned by PayPal you should cache the access token rather than getting a new one with every request. This plugin should be able to save the token securely on the server.

BTW, the v1 REST API was deprecated in 2019.

[nfourtythree]

Thank you. Are you able to provide a rough ETA (hours, days, weeks) so that we can help our clients decide a course of action to accept payments in the interim.

We hope to get something out tomorrow.

Thanks for looking into this. As mentioned by PayPal you should cache the access token rather than getting a new one with every request. This plugin should be able to save the token securely on the server.

Correct. However in the SDK from PayPal there is no apparent way to pass in an access token, and the way they deal with it uses a number of private properties and methods making it less straightforward to override. This is what we will work around to leverage the getting and settting of a cached token.

BTW, the v1 REST API was deprecated in 2019.

Looking at the current PayPal developer docs it appears that the authentication endpoint still references v1.

We used the official PayPal PHP SDK in hope that they would keep it up to date. However, as we can see that is not the case and have now deprecated the library.

[westhouseit]

Looking at the current PayPal developer docs it appears that the authentication endpoint still references v1.

We used the official PayPal PHP SDK in hope that they would keep it up to date. However, as we can see that is not the case and have now deprecated the library.

You're right, it's other APIs that moved to v2. I've actually had a horrible time with PayPal integration on another project. It's almost like they want developers to move away from their platform.

[nfourtythree]

Hi All

Thank you for your help and patience with this issue.

We have pushed out two plugin releases today as a fix for this.

• Version 1.3.3 for Commerce 3
• Version 2.1.1 for Commerce 4

Please update to the relevant version of the plugin and let us know if this fixes things. If you continue to get problems please comment on this issue so we can reopen it and investigate further.

Thanks!

[westhouseit]

@nfourtythree that's great news, thanks!

[westhouseit]

@nfourtythree sadly this doesn't appear to have fixed the issue.

So I went ahead and triggered the issue again, then downloaded the whole of the cache dir. Running a grep for the "PayPalAuth" key returns nothing in the 6008 files. Is the key not being cached or being cached somewhere else? We have no custom caching config.

It seems to me the token request & caching should be done client-side.

[nfourtythree]

Hi @westhouseit

If you search in those files for PayPalCheckoutSdk\Core\AccessToken (or parts of that string) do you get any results?

Trying to figure out if it is actually caching the access token.

Thanks

[westhouseit]

@nfourtythree No, sorry, no results.

[nfourtythree]

@westhouseit This seems like it isn't adding it to the cache.

Are you able to xdebug the process and see where it might be falling down?

Is this in both production and sandbox mode? Are you able to test with different client ID and secret values?

It could also be worth trying with a fresh install of Craft, Commerce and the plugin to eliminate any other external factors.

Obviously, I understand I am asking a number of questions here but just want to try and get to the bottom of this.

Thanks

[samueldraper]

Are you able to test with different client ID and secret values?

The same issue is occurring on three different websites, each which have different client IDs and secrets.

[nfourtythree]

The same issue is occurring on three different websites, each which have different client IDs and secrets.

Even after updating to the latest version of the plugin?

[samueldraper]

The same issue is occurring on three different websites, each which have different client IDs and secrets.

Even after updating to the latest version of the plugin?

Yes after the updates. Perhaps it’s helpful to clarify that @westhouseit and myself are collaborating on the same issues on the same sites.

[nfourtythree]

@samueldraper thank you for the clarification.

Have you had a chance to explore the other things I mentioned in my previous comment?

It seems, at least based on @westhouseit, that this could be a caching issue.

[westhouseit]

Are you able to xdebug the process and see where it might be falling down?

So, the debugging on local shows that the cache file is being saved. I'll see what I can do on production to get more details.

[westhouseit]

So, there's definitely been some order failures due to rate limiting and I've managed to get a little more details. The PayPal token is successfully cached for 9 hours, but the first request after the token expires results in RATE_LIMIT_REACHED. This repeats for about half an hour. A few hours later, which also happens to be after midnight, another token request succeeds and is cached.

It doesn't make sense that one request after 9 hours should result in rate limiting. I'll add some more debugging to try to narrow this further.