Rest API calls give 401 Unauthorized in response since Magento 2.4.6

[jorgb90]

Preconditions and environment

• Magento 2.4.6

Steps to reproduce

1. Create a product with a SKU with special characters, for example a forward slash.
2. Try to PUT or GET its stock or info through the REST API, see more info below in the comments.

Expected result

Working REST API.

Actual result

[Error message: 401 Unauthorized: The signature is invalid. Verify and try again.]

Additional information

Update 1 After debugging I found out that because of special characters in our SKU the signature comparison fails. Read below for more info and check https://github.com/magento/magento2/issues/37278#issuecomment-1478635403

Update 2 It should be a special character not allowed in an URL, so for example a forward slash which gets replaced by Hex. https://github.com/magento/magento2/issues/37278#issuecomment-1481245655

Updated above steps to be more in line with the problem.

Release note

No response

Triage and priority

• [X] Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• [ ] Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• [ ] Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• [ ] Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• [ ] Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant[bot]]

Hi @jorgb90. Thank you for your report. To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:

• @magento give me 2.4-develop instance - upcoming 2.4.x release
• For more details, review the Magento Contributor Assistant documentation.
• Add a comment to assign the issue: @magento I am working on this

• To learn more about issue processing workflow, refer to the Code Contributions.

  Join Magento Community Engineering Slack and ask your questions in #github channel. :warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting. :clock10: You can find the schedule on the Magento Community Calendar page. :telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

[m2-assistant[bot]]

Hi @engcom-Dash. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
• 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
• 4. Verify that the issue is reproducible on 2.4-develop branch
     Details

     - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
     - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
     - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[jorgb90]

It seems its related to the products endpoint only, need further debugging to confirm.

Update:

The 401's are received for the

https://www.domain.coml/index.php/rest/V1/products/:sku:

or

https://www.domain.com/index.php/rest/V1/products/:sku:/stockItems/1

endpoints.

Before 2.4.6 update this was a perfect 200 response, after update a 401 one.

[jorgb90]

Is there perhaps something changed for the decoding of special characters?

Update Then I would expect a 404 instead of a 401 but

"PUT https://www.domain.com/rest/V1/products/Z16K-006/stockItems/1"

works and gives 200 response and

"PUT https://www.domain.com/rest/V1/products/MNWD3N%2FA/stockItems/1"

does not and gives a 401 in response. Only difference seems to be the forwarding slash. Nothing has changed on the environment.

[jorgb90]

grep -rnw '/home/xxx/public_html/' -e 'The signature is invalid. Verify and try again.'
/home/xxx/public_html/vendor/magento/framework/Oauth/Oauth.php:207:            throw new AuthException(new Phrase('The signature is invalid. Verify and try again.'));

https://github.com/magento/magento2/blob/2.4.6/lib/internal/Magento/Framework/Oauth/Oauth.php#L207

This file is edited in comparison with Magento 2.4.5-p1:

https://github.com/magento/magento2/compare/2.4.5-p1...2.4.6#diff-4d3624e0045dd8880cca6a979be18ba743667c422c0fd98fc96e579e78e3c793

But dont understand why this would only give a 401 in certain conditions, like the examples above.

Update Its comparing Oauth headers and perhaps it fails when special characters are in play.

[jorgb90]

Can confirm that all integrations start working again when I let this comparison succeed

https://github.com/magento/magento2/blob/2.4.6/lib/internal/Magento/Framework/Oauth/Oauth.php#L206

So there is a problem with special characters and $requestUrl which give an invalid comparison and by that the check fails.

[m2-assistant[bot]]

Hi @engcom-Bravo. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
• [ ] 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
• [ ] 4. Verify that the issue is reproducible on 2.4-develop branch
  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
• [ ] 5. Add label Issue: Confirmed once verification is complete.
• [ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-Bravo]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @engcom-Bravo. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @engcom-Bravo, here is your Magento Instance: https://e71e346df51e726fff33fe598ba0adc8.instances-prod.magento-community.engineering Admin access: https://e71e346df51e726fff33fe598ba0adc8.instances-prod.magento-community.engineering/admin_0eba Login: a9237166 Password: 23d5df6f49eb

[engcom-Bravo]

Hi @jorgb90,

Thank you for reporting and collaboration. Verified the issue on Magento 2.4-develop instance and Magento 2.4.6 instance the issue is not reproducible. Kindly refer the screenshots.

In Magento 2.4-develop instance

In Magento 2.4.6 instance

In Both the cases the REST API is working fine.Kindly check your authorization tokens.

Please let us know if you are still facing any issue.

Thanks.

[jorgb90]

@engcom-Bravo Sorry I said back slash but it should be a forward slash. It should be a special character which isn't allowed in an URL and by that gets replaced with Hex. The SKU should be for example "MLY33N/A" and that becomes "MLY33N%2FA", so try to use "http://magento246.local/rest/default/V1/products/MLY33N%2FA" as URL.

[jorgb90]

@engcom-Bravo Update is already given.

[engcom-Bravo]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @engcom-Bravo. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @engcom-Bravo, here is your Magento Instance: https://e71e346df51e726fff33fe598ba0adc8.instances-prod.magento-community.engineering Admin access: https://e71e346df51e726fff33fe598ba0adc8.instances-prod.magento-community.engineering/admin_57cf Login: 272e1903 Password: 68373c8203e1

[engcom-Bravo]

Hi @jorgb90,

Thanks for your update.

Verified the issue on Magento 2.4-develop instance and Magento 2.4.6 instance the issue is not reproducible. Kindly refer the screenshots.

We have created a product with MLY33N/A and slash / is taken as - for a product.

REST API is working fine using PUT method in both the cases.

Magento 2.4-develop instance

Magento 2.4.6 instance

Kindly recheck the issue on Magento 2.4-develop instance and Magento 2.4.6 instance and check if any 3rd party extensions / modules enabled is causing this issue.

Thanks

[jorgb90]

@engcom-Bravo Not name of the product but SKU! Will explain it again:

1. Create a product with a SKU with a special character like forward slash, so for example SKU "MLY33N/A" and Name "Laptop".
2. Make a GET or PUT call to an endpoint like https://www.domain.com/index.php/rest/V1/products/:sku: so it becomes https://www.domain.com/index.php/rest/V1/products/MLY33N%2FA
3. This fails with the message "Error message: 401 Unauthorized: The signature is invalid. Verify and try again."

Only calls with these encoded special characters fail others succeed. This has nothing to do with 3rd party extensions or modules.

Update I now see you also did this wrong the first try.

Update 2 Here you go including screenshots. This request failed because of the encoded SKU.

And another request which succeeds because it has a not an encoded SKU.

[slavvka]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @slavvka. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @slavvka, here is your Magento Instance: https://e71e346df51e726fff33fe598ba0adc8.instances-prod.magento-community.engineering Admin access: https://e71e346df51e726fff33fe598ba0adc8.instances-prod.magento-community.engineering/admin_e0f7 Login: d923b56d Password: 88396b805886

[engcom-Bravo]

Hi @jorgb90,

Thanks for your update.

Verified the issue on Magento 2.4-develop instance and Magento 2.4.6 instance the issue is not reproducible. Kindly refer the screenshots.

We are getting 404 Not Found Response in both the instances.

We have created a product with name Laptop and sku MLY33N/A and we are hitting a GET Request.

Kindly let us know if we are missing anything.

Thanks.

[jorgb90]

@engcom-Bravo I don't know if it makes a difference but your adding "default" to the endpoint. According to the Magento documentation the endpoint "/rest/V1/products/:sku" should work and definitely not give a 404 Not Found.

Update Using NGINX in your setup or Apache? If Apache you also need to make sure you set AllowEncodedSlashes NoDecode

As per this

[slavvka]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @slavvka. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @slavvka, here is your Magento Instance: https://e71e346df51e726fff33fe598ba0adc8.instances-prod.magento-community.engineering Admin access: https://e71e346df51e726fff33fe598ba0adc8.instances-prod.magento-community.engineering/admin_014e Login: c7eec712 Password: 2e60b3f9f875

[engcom-Bravo]

Hi @jorgb90,

Thanks for your update.

We have Verified the issue on Magento 2.4-develop instance and Magento 2.4.6 instance without default in the endpoint still we are getting 404 Not Found Response in both the instances..Kindly refer the screenshots.

We have verified with without special characters we are getting 404 Not Found.

AS per https://httpd.apache.org/docs/2.4/mod/core.html#allowencodedslashes. if we allow slashes to decoded could potentially allow unsafe paths and security issues.use of NoDecode is strongly recommended as a security measure.

Kindly let us know if you are still facing any issue.

Thanks.

[jorgb90]

@engcom-Bravo How would you then access a SKU like "MLY33N/A" through the API? The only way is encoded, so you need to set Apache to NoDecode (this issue isn't there when using Nginx). And dont think that SKU's with special characters are rare, cause the SKU I gave as an example is an Apple SKU. They do it for every product they have and I can name a few brands more.

Here you see someone else mentioning it https://github.com/magento/magento2/issues/21179 and where they also mention to set NoDecode for Apache as a solution.

A 404 Not Found and rest the case isn't solving the issue so to speak. Please set your Apache accordingly (or use Nginx) and let's solve the "Error message: 401 Unauthorized: The signature is invalid. Verify and try again.". This problem wasn't there in Magento 2.4.5-p1 but it is now at Magento 2.4.6.

Update Apache isn't saying it's not recommending to use NoDecode its saying it's recommending it, otherwise it could lead to security issues because of wrong paths. Wrong paths like your 404's above occur because of decodes and by that are unsafe. Per default its decoding, so NoDecode needs to be set.

[slavvka]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @slavvka. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @slavvka, here is your Magento Instance: https://e71e346df51e726fff33fe598ba0adc8.instances-prod.magento-community.engineering Admin access: https://e71e346df51e726fff33fe598ba0adc8.instances-prod.magento-community.engineering/admin_6730 Login: 1dd38c70 Password: 1528cc2688e8

[magento-deployment-service[bot]]

Hi @slavvka, here is your Magento Instance: https://e71e346df51e726fff33fe598ba0adc8.instances-prod.magento-community.engineering Admin access: https://e71e346df51e726fff33fe598ba0adc8.instances-prod.magento-community.engineering/admin_6730 Login: 1dd38c70 Password: 1528cc2688e8

[engcom-Bravo]

Hi @jorgb90,

Thanks for your update.

We have set to AllowEncodedSlashes NoDecode

Still we are getting 404 NOT FOUND in the response.

Kindly let us if we are missing anythIng and provide any missing steps.

Thanks.

[jorgb90]

@engcom-Bravo I am not an Apache expert unfortunately, but the only way to get a 404 Not found is when decoding is still active, cause then it will rewrite

http://kumar.local/rest/V1/products/MLY33N%2FA

as

http://kumar.local/rest/V1/products/MLY33N/A

And thats of course a not existing path and by that gives a 404 Not Found.

A quick Google gave me that you probably need to set it inside a <VirtualHost> container as well. I think you are now doing it in global server config?

[engcom-Bravo]

HI @jorgb90,

Thanks for your update.

Verified the issue on Magento 2.4-develop instance and Magento 2.4.6 instance the issue is not reproducible. Kindly refer the screenshots.

We have set to AllowEncodedSlashes NoDecode inside a <VirtualHost> container.

Magento 2.4 develop instance

Magento 2.4.6 instance

We are able to retrieve the response.Rest API is working fine.

Thanks.

[jorgb90]

@engcom-Bravo Thanks. Very strange though. I will look into it further. The only thing we did was an upgrade from Magento 2.4.5-p1 to 2.4.6 and then it stopped working. So it must be something with the changes made to the framework or auth.

There are more things going wrong with slashes like for example https://github.com/magento/magento2/issues/37321

[engcom-Bravo]

Hi @jorgb90,

Thanks for your update.

verified the issue while upgrading from 2.4.5-p1 to 2.4.6 we are not able to reproduce the issue.Kindly refer the screenshots.

Kindly let us if still you are facing any issue.

Thanks.

[engcom-Bravo]

Hi @jorgb90,

We have noticed that this issue has not been updated for a period of 14 Days.
Hence we assume that this issue is fixed now, so we are closing it. Please raise a fresh ticket or reopen this ticket if you need more assistance on this.

Thanks.

[hostep]

@jorgb90: I just noticed this fix in Apache 2.4.57: https://github.com/apache/httpd/blob/5e5c0286bb3bd714f5c5b97f96189ed9ba589cdc/CHANGES#L9-L11

It sounds kind of related to this issue, so maybe it wasn't a bug in Magento, but in Apache that you ran into?

[jorgb90]

https://github.com/apache/httpd/blob/5e5c0286bb3bd714f5c5b97f96189ed9ba589cdc/CHANGES#L9-L11

Hi @hostep thanks for your update regarding this issue! We are running NGINX so this is unfortunately not the solution for us. We still haven't found a solution, for now still using this dirty method https://github.com/magento/magento2/issues/37278#issuecomment-1478635403

[mamsincl]

@jorgb90

index 898f70e2..af2fdc85 100644
--- a/vendor/magento/framework/Oauth/Oauth.php
+++ b/vendor/magento/framework/Oauth/Oauth.php
@@ -203,7 +203,19 @@ class Oauth implements OauthInterface
             $requestUrl
         );

-        if (!Security::compareStrings($calculatedSign, $params['oauth_signature'])) {
+        $calculatedSignWithEncodedPercentage = $this->_httpUtility->sign(
+            $allowedSignParams,
+            $params['oauth_signature_method'],
+            $consumerSecret,
+            $tokenSecret,
+            $httpMethod,
+            str_replace('%', '%25', $requestUrl)
+        );
+
+        if (
+            !Security::compareStrings($calculatedSign, $params['oauth_signature']) &&
+            !Security::compareStrings($calculatedSignWithEncodedPercentage, $params['oauth_signature'])
+        ) {
             throw new AuthException(new Phrase('The signature is invalid. Verify and try again.'));
         }
     }

a little bit less dirty than entirely passing the validation

[simonmaass]

we can confirm this issue - we are facing the exact same problem after updating to 2.4.6!

Can also confirm that with some rp_tokens the request works and on some it fails... here is one example:

Fails with customer that has rp_token: '0:3:lG1pD0EhYY/3XldvcYwU/ewp3Xnc6Oaz/l4O99LhYGBMDswnlB1ONZMfXoadqzdXBa9lCQ=='

[cykolln]

Can similarly confirm - using the same endpoint /products to request an item by sku containing encoded / . Testing with the same oauth creds using another product from the same store that does not contain the slash results in a successful response. Besides adjusting core code, have any other adjustments or workarounds worked for anyone? Slashes in skus are very common, we need to be able to accommodate them.

[jorgb90]

@engcom-Bravo Can this be reopened? Seems to be not an issue specific for us.

[jorgb90]

@jorgb90

index 898f70e2..af2fdc85 100644
--- a/vendor/magento/framework/Oauth/Oauth.php
+++ b/vendor/magento/framework/Oauth/Oauth.php
@@ -203,7 +203,19 @@ class Oauth implements OauthInterface
             $requestUrl
         );

-        if (!Security::compareStrings($calculatedSign, $params['oauth_signature'])) {
+        $calculatedSignWithEncodedPercentage = $this->_httpUtility->sign(
+            $allowedSignParams,
+            $params['oauth_signature_method'],
+            $consumerSecret,
+            $tokenSecret,
+            $httpMethod,
+            str_replace('%', '%25', $requestUrl)
+        );
+
+        if (
+            !Security::compareStrings($calculatedSign, $params['oauth_signature']) &&
+            !Security::compareStrings($calculatedSignWithEncodedPercentage, $params['oauth_signature'])
+        ) {
             throw new AuthException(new Phrase('The signature is invalid. Verify and try again.'));
         }
     }

a little bit less dirty than entirely passing the validation

This works as well. Thanks!

[jorgb90]

we can confirm this issue - we are facing the exact same problem after updating to 2.4.6!

Can also confirm that with some rp_tokens the request works and on some it fails... here is one example:

Fails with customer that has rp_token: '0:3:lG1pD0EhYY/3XldvcYwU/ewp3Xnc6Oaz/l4O99LhYGBMDswnlB1ONZMfXoadqzdXBa9lCQ=='

Interesting

Can similarly confirm - using the same endpoint /products to request an item by sku containing encoded / . Testing with the same oauth creds using another product from the same store that does not contain the slash results in a successful response. Besides adjusting core code, have any other adjustments or workarounds worked for anyone? Slashes in skus are very common, we need to be able to accommodate them.

Yes very common. Didnt find any other solutions then the ones above unfortunately.

[jorgb90]

When does this get fixed? Next release?

[boneio]

I've described why I think this is happening here: https://github.com/magento/magento2/issues/37913#issuecomment-1949008482

Thanks to the workaround suggested: https://github.com/magento/magento2/issues/37278#issuecomment-1572439283

[jorgb90]

Magento 2.4.7 and still got this issue.. @engcom-Bravo

[jorgb90]

2.4.7-p1 and still this issue not fixed.