Tons of subscribers to newsletter daily

[addison74]

This is a known issue in Magento. Please note there is no email subscription input text box as I removed it from the frontend template. However, there are around 50 new subscribers every day. In forums I read there are people with hundreds of subscribers daily.

I managed to collect from the webserver access log all these IP addresses with Fail2Ban filters and I banned them. In one a week there are 3200 IP addresses. Here is the format of the request in access log:

109.1.105.79 - - [26/Oct/2020:10:28:22 +0200] "POST /newsletter/subscriber/new/ HTTP/1.1" 302 577 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"

Maybe you should fix this issue once and for all in OpenMage that the Magento team didn't want to fix over the years. There are a few extensions out there that are limited to the existence of the input text box and not to such requests made directly from a script.

Blocking these requests is crucial because there is a high chance that an email server such as Yahoo will automatically block an email or IP address for a long time until the human operator can remove it. Although it seems harmless at first glance, in time it can lead to a lack of communication with customers if the store's messages are considered SPAM.

[colinmollenhour]

Thanks for your issue reports, this and the others are all good common-sense suggestions. However, OpenMage is very much a community project so we don't really have resources to put on issues unless it is a critical security vulnerability or bug. For general improvements if you want to see it done the best way to make sure it gets done is to roll up your sleeves and submit a PR.

If you don't use the built-in newsletter functionality I would suggest disabling the module altogether via app/etc/modules/Your_Custom_File.xml, this will cause that URL to return a 404 and all newsletter features to be deactivated.

[addison74]

@colinmollenhour: I hope this project will attract more people who love Magento 1 platform. I agree critical security vulnerabilities or serious bugs are vital but without adding functionality features to this project it will remain at the maintenance level. For some reported issues I already provided solutions more will come in the near future. My only hope is others with more experience than mine to join forces.

[seansan]

@ADDISON74 Maybe this helps

• We use Fishpig Nobots protection has several methods of protection. Adding a similar method as this to core would not be uncommon. But you can buy it and be rid of the spam.

• Also we found out there was 1 custom form without a formkey : making it even more simple to mass entry email addresses

• Lastly, I do agree that some kind of "shared spam or hack ip" list would not even be such a bad idea. If more than 2 domains report a bad IP, then mark it bad for all (based on directory scanning, code injection attemptes, or form manipulation)

[addison74]

Using this command in Terminal I am able to add as many email addresses as I want into the database:

curl 'https://www.mydomain.com/newsletter/subscriber/new' --data 'email=johndoe@gmail.com'

The same thing is happening for creating new customer account using more data fields.

If I have the bad inspiration to set the options in Backend to confirm a subscription or an account Magento becomes a spammer. With Fail2Ban we catch daily around 60 new IP addresses based on a Regex filter. This is how the webserver log looks for such of requests:

"GET /customer/account/create/ HTTP/1.1" 200 82432 "-" "-"
"POST /newsletter/subscriber/new/ HTTP/1.1" 302 3874 "-" "-"

I gave it a try to an extension named HoneySpam but I did not get any positive results. I am still able to use curl command without any issue. As I understand they are using a hidden input field in the forms but what happens if the controller is used directly based on the URL? Or maybe I am doing something wrong?

@seansan Please give me more details how you implemented the fromkey. Just adding it to the template won't change anything some changes in the code should be made.

[seansan]

Ahum .. I thought this was blocked but I see also this work using the curl command ... scares me!

No solution yet then

On Fri, Nov 13, 2020 at 3:32 PM ADDISON notifications@github.com wrote:

Using this command in Terminal I am able to add as many email addresses as I want into the database:

curl 'https://www.mydomain.com/newsletter/subscriber/new' --data 'email=johndoe@gmail.com'

The same thing is happening for creating new customer account using more data fields.

If I have the bad inspiration to set the options in Backend to confirm a subscription or an account Magento becomes a spammer. With Fail2Ban we catch daily around 60 new IP addresses based on a Regex filter. This is how the webserver log looks for such of requests:

"GET /customer/account/create/ HTTP/1.1" 200 82432 "-" "-""POST /newsletter/subscriber/new/ HTTP/1.1" 302 3874 "-" "-"

I gave it a try to an extension named HoneySpam but I did not get any positive results. I am still able to use curl command without any issue. As I understand they are using a hidden input field in the forms but what happens if the controller is used directly based on the URL? Or maybe I am doing something wrong?

@seansan https://github.com/seansan Please give me more details how you implemented the fromkey. Just adding it to the template won't change anything some changes in the code should be made.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OpenMage/magento-lts/issues/1287#issuecomment-726796902, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAE7I22R7MOEI4PLQODDICLSPU7PJANCNFSM4TCD5KFQ .

[addison74]

Without fixing Magento code there are two options for the moment:

1. Do not activate confirmations for accounts and subscribers into the Backend. But you risk to send regularly emails to people who did not subscribe till they really upset to report you. Your email server IP will be blacklisted for sure. Yahoo likes sending messages in Spam folder.

2. You activate confirmations for accounts and subscribers into the Backend and catch every direct request from controller as I do with Fail2Ban. Daily I spend a few minutes to clean the dirt. Comparing with solution from point 1 only one time an email is sent to an address the one with confirmation. You also can control the subscriptions and delete those who came from a bot. If someone is clicking on a link and you already deleted the records from DB he will understand from redirection that you did not subscribe him (if he doesn't check the email headers).

Anyone with minimal knowledge of programming can fill the subscribers table with junk information using that command in a bash script in an endless loop.

With GDPR in Europe Magento Newsletter module needs improvement. At least a new column into the table to record the IP address which made the request.

[magenx]

in magento 2 the same issue since day one, they call it OK, because API is meant to be this way :)

https://gist.github.com/magenx/5bbc66555a0c90435c108a500c170ae3

[addison74]

@magenx: That word you go to bed happy in the evening and in the morning you pull your hair out when you see the website down. Especially on a Black Friday. If the IPs are not blocked immediately with a script like the one you created and a few computers running it, you fatten them well from the whole database. I am "terrified" that so far no solution has been found in either M1 or M2 for years.

[seansan]

1. We would start by adding a formkey in the form? validate it upong data submission? https://magento.stackexchange.com/questions/119942/form-key-value-in-template-and-controller

2. add captcha 2nd? https://magento.stackexchange.com/questions/70351/add-captcha-to-subscribe-phml

[addison74]

I vote for the variant 1. Once the path is requested to check if there is a formkey field before adding the records in database and sending any confirmation emails. If formkey is missing this request is not coming from form and should be rejected with an error and optional written in log. Fail2Ban can use this log to ban the IP for a while. If the formkey exists then it can be checked as value.

This should be achieved for Newsletter and Customer Account. I did not check for Contact form if the controller can be accessed with curl command and some data in the same way.

[seansan]

Part I Newsletter, can someone review and create PR? (sorry cant do it here)

Add to Mage_Newsletter_SubscriberController

    public function newAction()
    {

        if (!$this->_validateFormKey()) {
            Mage::throwException($this->__('Validation has failed.'));
            $this->_redirectReferer();
            return;
        }

Add to 2 templates here (dont forger to check theme files) frontend/base/default/template/newsletter/

    <form action="<?php echo $this->getFormActionUrl() ?>" method="post" id="newsletter-validate-detail">
    <input name="form_key" type="hidden" value="<?php echo Mage::getSingleton('core/session')->getFormKey() ?>" />

PART II

Idem for the new customer registration creation

I am presuming that newAction is only triggered from the frontend by a form and not in some other processes during checkout and by API and others. If so then we need to create a new frontend method that does the validation and exclude the existing one from being accessed from he frontend /w post details

PART III

Do the same for contact form?

[addison74]

@seansan: It is a good start and I appreciate you want to solve this issue.

We can go further creating an option in System Configuration -> Customers -> Newsletter to enable logging in a file for all failed validations (Yes/No option). In this way this file can be used with Fail2Ban and is more reliable than webserver log (like I do for the moment).

Same option should be created for Customers -> Customer Configuration -> Create New Account Options.

[addison74]

PART III

Do the same for contact form?

If the controller can be accessed directly with some data it should be changed too

[addison74]

I implemented this change in a production website. I continue to keep tracking the IP's that access the controller directly and banning them. In an hour there are 2 banned IP's and no records in newsletter_subscriber table. So far so good. From now on Magento won't send emails for confirmation based on these requests. An improvement to the code.

[addison74]

... an [improvement] label should be added to this issue.

[addison74]

We should continue testing this solution. Some requests skip even the formkey checking in controller. Here is one request from my webserver log which was recorded into the database:

45.72.31.243 - - [14/Nov/2020:18:32:38 +0200] "GET http://www.mydomain.tld/ HTTP/1.1" 302 194 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"
45.72.31.243 - - [14/Nov/2020:18:32:40 +0200] "GET / HTTP/1.1" 200 14726 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"
45.72.31.243 - - [14/Nov/2020:18:32:43 +0200] "POST /newsletter/subscriber/new/ HTTP/1.1" 302 578 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"
45.72.31.243 - - [14/Nov/2020:18:32:43 +0200] "GET / HTTP/1.1" 200 14588 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"

According to the mail server log Magento sent the confirmation email to the email address used by the bot.

Using curl command as I tested before your solution it is working fine. It seems this request somehow got a valid formkey and validate it in controller. There is no record in exception.log file. I will remove the form from the page for two days to see what is happening.

[addison74]

With your modification for Newsletter controller and the template content removed from frontend there is no subscriber in 48 hours. Once I enabled the form I got in the mesh 3 requests.

In order to partially fixing once and for all this issue we should identify all controllers that allow directly access to record data into the database which are skipping the forms like Newsletter. Probably Contact and New Customer Account forms can be used with curl command too.

Once the controllers are fixed a second layer of protection is needed - Captcha. Magento has an old style built-in only for a part of the forms. I recommend as follows:

1. Hackathon Magento Honey Spam
2. MageComp Magento Google reCAPTCHA

Both are free and up to date. I prefer Honey Spam because I don't want the visitors to spend time in solving visual puzzles. Sometimes this is annoying especially when the questions are not cleared at all.

If a third layer of protection is needed I recommend Fail2Ban.

[addison74]

Hackathon Magento Honey Spam is good but it doesn't solve the problem entirely.

If anyone is using this extension I recommend changing the hidden honeypot field name in backend from the default url to name. There are bots which are checking the input box names. If it is a common word then they fill it up. In this case the extension will block the request 100%.

Why it is not a perfect solution like using reCAPTCHA? Because there are bots which are filling up only the input text boxes having as names something related to email. Email word is a common one and well known for Magento. In this case the bot can skip all the others hidden input text boxs.

In conclusion, we need to patch all controllers that allow passing data to the database without checking the existence of a formkey in the forms.

[addison74]

I will close this issue and post it in Discussions section.