Closed eurobeautiful closed 3 years ago
@wjpbill It's probably not a good idea to share the rules setup publicly to give attackers clue how to circumvent the protection. Feel free to DM me on Twitter @SergiiShymko or LinkedIn @sshymko.
Currently Rate Limit is the only solution available that will block the attackers. If your site doesn't have Cloudflare then you should install and setup rules to block it.
If you are on the cloud server then you should look into setting up rate limit thru Fastly.
https://github.com/fastly/fastly-magento2/blob/master/Documentation/Guides/RATE-LIMITING.md
I hope it works.
The recaptcha feature does work. As someone else has already pointed out, the Swiss up labs module works and has built in support for checkout. Just enable it when the attack starts, Walt some time then turn it off and repeat
Sent from my iPhone
On Sep 22, 2020, at 5:07 PM, Kinex Media notifications@github.com wrote:
Currently Rate Limit is the only solution available that will block the attackers. If your site doesn't have Cloudflare then you should install and setup rules to block it.
If you are on the cloud server then you should look into setting up rate limit thru Fastly.
https://github.com/fastly/fastly-magento2/blob/master/Documentation/Guides/RATE-LIMITING.md
I hope it works.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
@wjpbill It's probably not a good idea to share the rules setup publicly to give attackers clue how to circumvent the protection. Please DM me on Twitter @SergiiShymko.
@sshymko
Twitter won't let me DM you,, just says you can't be messaged. Maybe you could start the message @wjpbill
If someone could help me along with the Rules for Rate limit that would be appreciated, struggling to get them to work
If someone could help me along with the Rules for Rate limit that would be appreciated, struggling to get them to work
Assuming you mean CloudFlare Rules... There is not much to the rule really... You can email me via my contact details on my profile and I will send you what you need.
@scottbolasevich Not a helpful comment. M2 is plenty good and powered in Commerce Cloud it's pretty awesome. Plenty of carts are susceptible to carding attacks, not just Magento. As noted above, folks should be contacting their reps for help as I did. If you raise the flag to those that can help you, things can get done. Magento announced a upcoming patch for recaptcha in checkout and api.
I will note we had to developer recaptcha for checkout with BrainTree on our own while we wait on an official patch. That's my only gripe, I wished they could have turned something quicker to have saved me the dev hours. If I didn't we'd still be getting hammered.
Having recaptcha at the API level, rate limiting on the api url and payment gateway fraud rules indeed work. My team even went one step further and developed auto IP banning for different classes of networks. Attacker will try once, get a recaptcha and move on. I get piece of mind auto banning the subnet they are coming from. I did share our Fastly mods with Magento and gave them permission to resue it. Its a pretty awesome feature to be able to insta block api checkouts if you're not supporting them. Blocking multiple network octets helps the whack-a-mole as well.
If anyone else is interested in the Fastly auto ban feature that works with recaptcha please lobby your account reps. Refer them to this thread. The more people that ask for it, the more inclined they'd be willing to include the feature. I'd love to see this included in core.
Remember folks security is always a layered approach!
yea. cloudflare, it only save you from dumb bots, but even now bots know how to get your real ip address first, and then attack. so if you are using any cloudfare service alike you need to either stop responding without cf header or filter connections on your server itself. waf mods for apache or nginx works better than any cloudflare...
@Magenx, you are correct. Hackers now using a pool of IP addresses and rate limit is failing to stop these attacks. Can you please share the waf mode rules which worked for you? My skype is karan.kinex.
Thanks, Karan
On Oct 14, 2020, at 9:46 AM, MagenX notifications@github.com wrote:
yea. cloudflare, it only save you from dumb bots, but even now bots know how to get your real ip address first, and then attack. so if you are using any cloudfare service alike you need to either stop responding without cf header or filter connections on your server itself. waf mods for apache or nginx works better than any cloudflare...
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.
@magenx Attackers cannot in principle get your real IP if it's not exposed anywhere and Cloudflare is proxying all the traffic. If they manage to get the real IP somehow, it wouldn't be a problem at all as the infrastructure should be configured to block all traffic other than Cloudflare IPs/subnets. For example, in AWS that can be implemented in a Security Group on a load balancer level.
to get real ip address is the easiest task today,
the fact that even magento commerce/cloud hosting just using Fastly without any other protection, you can connect directly to bypass Fastly and do whatever you want.
im saying is that this issue is not a magento fault, (unless they are going to build some admin settings to control API requests) and hiding behind Fastly or Cloudflare is useless without proper understanding and proper server configuration.
For Commerce Cloud customers, check with your reps about jumping onto Origin Cloaking ASAP. Its a feature rolling out to everyone soon. I was just rolled out in the past two weeks to stage/prod. It was seemless for as as we don't use direct IPS to connect.
@sshymko - I sent you the invite on Skype and Linkedin. Can you please share the rules you used thru skype?
@kinexm haven't received anything on either LinkedIn or Skype. My Skype is s.shymko and LinkedIn sshymko. Would prefer LinkedIn though.
I sent you the invite a few days ago. It under @simantsharma. Thanks, Karan
KARAN SHARMA Development ManagerKinex Media Inc.416-907-4030 Ext 402 |416-907-0635 karan@kinexmedia.comwww.kinexmedia.com25 Watline Avenue, Suite 302, Mississauga
On Wed, Oct 14, 2020 9:00 PM, Sergii Shymko notifications@github.com wrote:
@kinexm haven't received anything on either LinkedIn or Skype. My Skype is s.shymko and LinkedIn sshymko. Would prefer LinkedIn though.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
unless they are going to build some admin settings to control API requests
This is exactly what is required from Magento... API control settings that would allow sessions / carts to be invalidated based on configurable parameters.
This issue shows clearly illustrates how the API is vulnerable. All other discussions are just security semantics that differs in relevancy for each consumer dependant on environment. Sadly I think most of the input has assigned this thread to be ignored.
@Flipmediaco I agree lots are security steps. Your suggestion about a setting to invalidate cart is great. I encourage you and everyone to share that info with their reps and request calls with security team members to review these. From past experience this summer they are listening and got on calls with me to talk through options and a path forward.
perfect solution is to setup rate limiter
https://www.nginx.com/blog/rate-limiting-nginx/
capture cartid
and feed offensive IPs into firewall or into deny
configuration block in nginx
.
attacker most likely generate huge amount of 400
error code in short time, this is even easier to ban.
Had 3300+ attempts in less than 3 hours this morning. Had to disable the payment system. 2.3.5-p1 with a 3rd party payment gateway. reCaptcha V3 is running, but not in checkout of course.
From what I have read there is nothing that can be done? Is Magento dev team attempting to fix this?
@Grumag are you cloud commerce or on-prem?
Read through some of the solutions I offered above. Rate limiting, payment gateway limiting/fraud controls. ReCaptcha is coming to the checkout and api in an upcoming release. ReCaptcha at the API layer is the godsend here. We have to built it ourselves this summer. Also with a firewall you can block requests coming to the API url.
If you have an enterprise licence, reach out to support and request help along with your account rep. There are some things they'll assist and guide you on. I'm a broken record here on this thread. The more people that talk to their reps, raise their hands for escalation will help. I was on calls with multiple team members to work through my sites. They helped me with firewall rules and Fastly. Hopefully your payment gateway has some fraud tools that will help too.
Seems like the simplest solution would be:
bind the session cookie to the user's IP address, and flush the session and redirect if the IP changes. A few users on dynamic IPs would lose their cart contents/login session when their ISP's DHCP lease renews, but that's a monthly or less frequent occurrence AFAIK (thought it may be different outside the USA) --- I think this can be accomplished in admin via the Validate REMOTE_ADDR setting - https://docs.magento.com/user-guide/stores/security-session-validation.html --- if your load balancer/WAF/HTTP server is not configured correctly, Magento may not actually receive the real user IP, in which case you need to fix that situation first
rate limit checkout POST operations for a single cart, like enforce a 10 second delay between subsequent requests and extend that timeout after the third request (I'm guessing it's a very small minority of users who would need to try more than three credit cards to find one that works)
On Wed, Oct 21, 2020 at 1:23 PM Nick M notifications@github.com wrote:
@Grumag https://github.com/Grumag are you cloud commerce or on-prem?
Read through some of the solutions I offered above. Rate limiting, payment gateway limiting/fraud controls. ReCaptcha is coming to the checkout and api in an upcoming release. ReCaptcha at the API layer is the godsend here. We have to built it ourselves this summer. Also with a firewall you can block requests coming to the API url.
If you have an enterprise licence, reach out to support and request help along with your account rep. There are some things they'll assist and guide you on. I'm a broken record here on this thread. The more people that talk to their reps, raise their hands for escalation will help. I was on calls with multiple team members to work through my sites. They helped me with firewall rules and Fastly. Hopefully your payment gateway has some fraud tools that will help too.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/magento/magento2/issues/28614#issuecomment-713775440, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM7FFR7ZEAHVFDLSG2KFXDSL4RLDANCNFSM4NYKEZMA .
I'm running in commerce cloud I can confirm that "Validate REMOTE_ADDR = YES" doesn't help. That was one of the things that support suggested, however it didn't stop. Totally enable it nonetheless, but it doesn't help kill the carding attacks.
Did anyone notice that 2.3.6 security patch has the following:
CAPTCHA protection has been added to the following product areas:
Place Order storefront page and REST and GraphQL endpoints Payment-related REST and GraphQL endpoints. CAPTCHA protection for these additional pages is disabled by default. It can be enabled on the Admin in the same way that other pages covered by CAPTCHA are. This protection has been added as an anti-brute force mechanism to protect stores against carding attacks.
https://devdocs.magento.com/guides/v2.3/release-notes/open-source-2-3-6.html
yes, 2.3.6 it was released 8 days ago,
im playing with kinda nginx limiter + firewall setup
needs some finetuning and probably add some map
and set
variables to test requests
also combining with proxypass to catch status 400, etc
http {
....
limit_req_zone $cartId zone=cartid:35m rate=10r/m;
....
server {
....
location ~ /V1/guest-carts/(?<cartId>.+)/payment-information {
if ($cookie_form_key = "") { return 429; }
limit_req zone=cartid;
limit_req_status 429;
try_files $uri $uri/ /index.php$is_args$args;
error_log /var/log/nginx/payment.log
}
....
}
}
Did anyone notice that 2.3.6 security patch has the following:
CAPTCHA protection has been added to the following product areas:
Place Order storefront page and REST and GraphQL endpoints Payment-related REST and GraphQL endpoints. CAPTCHA protection for these additional pages is disabled by default. It can be enabled on the Admin in the same way that other pages covered by CAPTCHA are. This protection has been added as an anti-brute force mechanism to protect stores against carding attacks.
https://devdocs.magento.com/guides/v2.3/release-notes/open-source-2-3-6.html
Maybe don't get too excited...
M 2.3.6 Known Issues: https://devdocs.magento.com/guides/v2.3/release-notes/open-source-2-3-6.html#known-issues
Issue: The new CAPTCHA feature for checkout does not work as expected on the Place Order page when using third-party payment providers. Merchants running Magento 2.3.6 or 2.4.1 who have enabled CAPTCHA protection on the Place Order storefront page will see this error when checking out using a third-party payment provider such as PayPal: Please provide CAPTCHA code and try again. A fix for this issue be available in the next few weeks, and will be included in our next quarterly patch (Q12021). Please contact Support for additional information.
Rate limiting should be part of the security mix, but it's not the solution to this problem.
In the worst attacks, we were hit by more than 500 different IP addresses, all trying the same cartid. This happened in spite of all the following settings activated (set to Yes): Validate REMOTE_ADDR Validate HTTP_VIA Validate HTTP_X_FORWARDED_FOR Validate HTTP_USER_AGENT
yes, the limit against <cartid>
as im pretty sure this request is remote injecting CC numbers into API, you cut both cookie and cartid, so attacker bot will not be able to use checkout.
Magento team should probably release a standalone plugin to deal with this. Many merchants cant just get up and go and install new Magento 2 version.
yes, the limit against
<cartid>
as im pretty sure this request is remote injecting CC numbers into API, you cut both cookie and cartid, so attacker bot will not be able to use checkout.
Thanks for your NGINX rules! I have tested them and they work great when I was testing with siege.
to get real ip address is the easiest task today,
the fact that even magento commerce/cloud hosting just using Fastly without any other protection, you can connect directly to bypass Fastly and do whatever you want.
im saying is that this issue is not a magento fault, (unless they are going to build some admin settings to control API requests) and hiding behind Fastly or Cloudflare is useless without proper understanding and proper server configuration.
Yes original server IP can be easily found, but you can set up your server to allow connections only to Cloudflare IPs. This way even if bad guys want to connect to your server, they will be denied and will need to connect via Cloudflare only.
Cloudflare IP list: https://www.cloudflare.com/ips/
I've created some fail2ban rules around this. They assume the rest api is not in use. Here is my writeup about it. But in short here:
vi /etc/fail2ban/filter.d/restapi.conf
[INCLUDES]
before = restapi.conf
[Definition]
failregex = ^ - .* "POST.HTTP." 400 .*$
vi /etc/fail2ban/filter.d/restapi.conf
[restapi]
enabled = true
port = http,https
filter = restapi
logpath = /var/www/vhosts/yourdomain.com/logs/access_ssl_log
bantime = 86400
findtime = 1200
maxretry = 5
service fail2ban restart
fail2ban-client status restapi
script to place forged order into magento 2.4.1 this is not new, but raises many questions
https://gist.github.com/magenx/bdc56bf568caa3c23b2217055aef17b2
@magenx I tried your nginx rules, but we were getting kicked back during the checkout process to the cart page when trying "real" test orders. So we had to revert.
I saw in your repo you had added them. I'm wondering if there were any material changes that helped from this ticket. I've added:
limit_req_zone $cartId zone=cartid:35m rate=10r/m;
location ~ /rest/default/V1/guest-carts/(?<cartId>.+)/payment-information {
if ($cookie_form_key = "") { return 403; }
limit_req zone=cartid;
limit_req_status 429;
try_files $uri $uri/ /index.php$is_args$args;
error_log /var/log/nginx/payment.log;
}
Did I miss something?
@magenx I tried your nginx rules, but we were getting kicked back during the checkout process to the cart page when trying "real" test orders. So we had to revert.
I saw in your repo you had added them. I'm wondering if there were any material changes that helped from this ticket. I've added:
limit_req_zone $cartId zone=cartid:35m rate=10r/m;
location ~ /rest/default/V1/guest-carts/(?<cartId>.+)/payment-information { if ($cookie_form_key = "") { return 403; } limit_req zone=cartid; limit_req_status 429; try_files $uri $uri/ /index.php$is_args$args; error_log /var/log/nginx/payment.log; }
Did I miss something?
I think you might need to add burst and delay to this (only works on newer versions of NGINX).
Burst allows multiple requests to succeed before throttling.
location ~ /rest/default/V1/guest-carts/(?
I've tested with siege and without burst this throttling looks something like this:
accept deny deny deny deny accept deny deny deny deny accept
With burst it looks like this:
accept accept accept accept accept deny deny deny deny accept
You can test with siege like this: siege -v -c 1 -i -t 3M http://magento.test/rest/default/V1/guest-carts/CART_ID/payment-information
Remember to remove the no cookie blocking before testing.
@pmonosolo you don't need to push more than one request to payment-information, as you set payment only once, in some cases two is max, but for human you dont push in per second, any subsequent requests are bad. first one looks correct behavior..
@pmonosolo you don't need to push more than one request to payment-information, as you set payment only once, in some cases two is max, but for human you dont push in per second, any subsequent requests are bad. first one looks correct behavior..
Thanks.
BTW carding continues with Guest Checkout turned off. They just created a user and tested again today.
The URL they are using to spam is: POST /rest/default/V1/carts/mine/payment-information HTTP/1.1
@pmonosolo you don't need to push more than one request to payment-information, as you set payment only once, in some cases two is max, but for human you dont push in per second, any subsequent requests are bad. first one looks correct behavior..
When I was testing your solution with siege sometimes the first request would be rejected. I dont know why that was happening. As soon as I added burst it allowed for the first transaction to go through.
hello
Using cloudflare and setup rule to block unwanted ip resolves this issue: You can use following url patern
*`/rest//V1/guest-carts//payment-information`**
then set same IP exceeds 5 time per minute
I tested and see thatc the whole checkout process of a customer just call */rest/*/V1/guest-carts/*/payment-information
1 time so for a single IP, 5 time per minute is marked as attacker, event you can set this number to a lower value like 4 or 3
The recaptcha feature does work. As someone else has already pointed out, the Swiss up labs module works and has built in support for checkout. Just enable it when the attack starts, Walt some time then turn it off and repeat
@aliomattux You're confirming that the Swissup labs extension protect your checkout, but do you know if your attacks are processed through the REST API? Because Swissup labs says to us that they are not sure that their extension can protect this type of attacks and you are almost sure that our attacks don't hit the checkout page but goes straight with the API (no traffic spike in GA). Thanks you
I believe all checkout requests use rest. Look at the access logs for your web server. Yes I’m sure it works
Sent from my iPhone
On Nov 17, 2020, at 6:57 AM, jeckojim notifications@github.com wrote:
The recaptcha feature does work. As someone else has already pointed out, the Swiss up labs module works and has built in support for checkout. Just enable it when the attack starts, Walt some time then turn it off and repeat
@aliomattux You're confirming that the Swissup labs extension protect your checkout, but do you know if your attacks are processed through the REST API? Because Swissup labs says to us that they are not sure that their extension can protect this type of attacks and you are almost sure that our attacks don't hit the checkout page but goes straight with the API (no traffic spike in GA). Thanks you
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
One of our Clients had this issue, we had to disable guest checkout, there was no workaround in 2.2.x
@sb-nick the attack stopped when I set the following to "Yes"
Validate REMOTE_ADDR Validate HTTP_VIA Validate HTTP_USER_AGENT
The only problem is, my website now keeps redirecting to the homepage after a few minutes of inactivity. I tested on the staging website, and it throws the following error "Exception #0 (Magento\Framework\Exception\SessionException): The "remote_addr" session value is invalid. Verify and try again."
I don't understand how the "remote_addr" keeps on changing. Thoughts?
@farhan-saeed17 did you do a full system cache clear of everything? I didn't experience that with my instance. Are you running commerce cloud or on prem commerce?
Has anyone tried 2.3.6 with patch MC-38033_2_3_x-p1CAPTCHA_COMPOSER which apparently fixes reCaptcha implementation for checkout?
I just experienced my first attack and I am on 2.4.1. What a bad timing since this is our most busiest month of the year and we are asking customers to call in for manual payment!! I read through the whole thread and I am so discouraged!!!! :(
One of our Clients had this issue, we had to disable guest checkout, there was no workaround in 2.2.x
Disabling guest checkout only works briefly, as the carder simply creates an account manually and then uses that to run the carding attack. You MUST have a captcha solution enabled on the create/login account function. I can also say, as much as it pains me to have to, in our Cloudflare WAF settings disabling China (N Korea) and Russia ASN's for checkout flow drastically reduced the frequency of carding attacks.
I just checked the release notes for 2.4.1 and carding attacks was one of the issue they have fixed (https://prnt.sc/vv0nak). So how come I am still getting attacked?!
Is everyone using braintree in this thread? I'm only seeing carding attacks across merchants using braintree. We have not seen our other merchants (auth.net) get hit. Probably because we're using a third party auth.net module and it maybe doesn't have high enough saturation for the hackers to script around when so many magento/braintree sites exist?
@chichitsai0517 have you reached out to support to have them take a look at your system? Mage support jumped and helped me review and configure fastly rules and other items. I encourage everyone to reach out to support and their Magento customer reps to ask for help. Don't go at it alone, they will assist.
I'm guessing the attack your having is coming through the API layer. That new patch should have recaptcha for the API layer. In addition, we also turned on recaptcha on our checkout pages to so its on in both spots. Combined with the Fastly rules mentioned above we haven't had issues in 3 months.
Recap of what we did: 1) Set these to yes: Validate REMOTE_ADDR Validate HTTP_VIA Validate HTTP_USER_AGENT 2) Enabled recapctha at checkout on the storefront side 3) Enabled recapctha at the api layer 4) Added two fastly firewall rules for the urls - noted above in this tread 5) Rate limiting with fastly plugin for the api and checkout urls - noted above in this thread 6) In the payment gateway we setup up other fraud rules to block users transactions after x amount of time. Work with your processor on fine tuning. BrainTree support got on the phone with me to configure.
Another thing we did proactively was build an auto IP ban mini module that reads sever logs and auto bans IPS and their class network range if x entries are found in the logs to the checkout api url. However with the two recaptcha layers on I'm not seeing as many bans ATM. I did donate this code to Magento support for review. I begged them to implement this for everyone to leverage and make it a part of the Fastly module for commerce cloud. Please feel free to lobby them to include it.
@sb-nick Thank you I am actually not with Braintree but with Moneris. How did you get the Recaptcha to work in frontpage Checkout? I tried in Customer>Customer Configuration>Captcha>Enable CAPTCHA on Storefront (Set to yes to Checkout/Placing Order). However it does nothing in the frontend checkout. If you are talking about google recaptcha on checkout frontend, there is not option for that in my magento admin. They have other pages but not for checkout. Can you tell me what is recaptcha at api layer?
My store has been the target of several carding attacks in the last few weeks, growing in intensity and sophistication. Thousands of credit cards (presumably stolen numbers) are "tested" using a single guest cart on my store.
Initially we were manually blocking IP addresses after receiving the Payment Transaction Fail email notices, and increased Cloudflare security level. When the attacks became more frequent, created a custom fail2ban filter that would ban IP addresses having several POST requests to
/rest/<view>/V1/guest-carts/<cart_id>/payment-information
.Then the attacks started to come from dozens of IP addresses. At that point we noticed that ModSecurity wasn't working, and once we got that fixed, they have mostly stopped.
Here's a small sample from the Apache logs... they continue with hundreds of different IP addresses:
This last escalation is what makes me think that there may be a vulnerability in Magento. Is it normal for Magento to accept activity from the same session, from different IP addresses? As you can see, all the POST requests use the same cart ID, and my guess is that there are additional cookies and tokens necessary for the requests to be accepted and processed by Magento.
You may have noticed that the log includes 3 different IP addresses on each line. This is because we have a multi-proxy setup: Cloudflare > nginx > Varnish > Apache. Maybe Magento isn't catching the real remote user IP address but instead using 127.0.0.1?
Preconditions (*)
Other conditions that may or may not be related:
Steps to reproduce (*)
/rest/<view>/V1/guest-carts/<cart_id>/payment-information
from different IP addresses.Expected result (*)
Actual result (*)
Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.