Closed merpdotcom closed 9 years ago
i just checked the registration form of the site http://www.spokanerpg.com/@@register
it seems your captcha is always asking the same question: "Say farewell in Tolkien's Elvish"
although i don't have an idea (and - shame on me - could not come up with a working translation within 1 minute) i guess other people can do so. as you're asking the same question everytime on can easily start to run automated registrations by providing the correct answer for your question.
the key with captchas is that they need to be random
General discussion in IRC included suggestion to use collective.recaptcha with version 2 recaptchas, but beyond that that it would be nice to change the registration process so that emails went out FIRST before users are created.
I had many different Q/A responses. I had just temporarily dropped it to one while working with the IRC folks. And I have been completely making up utterly obscure, including just random pwgen-based passwords that don't fit the question, just for testing. It seems more that captcha, whether old captcha, recaptcha, or norobots, are seeming to be simply bypassed. I am getting recaptcha v2 setup right now for testing, but I don't know that will make any difference. But I am trying anything. For testing sake for you guys, I have re-enabled self-registration, and added back a variety of Q/A examples. Thanks for trying to help.
@merpdotcom you're currently not using version 2 of recaptcha. That seems to be the most promising suggestion at this point.
...It'd be nice if we could get someone who hosts websites that have self-registration chime in and let us know what works best. I've never hosted a site that allowed self-registration.
I've been hosting self-registration for dozens of community sites since the late 90s. In 2003/4 I switched them over to Zope-Plone. It had some challenges with registration spam, forum spam, contact spam (but was much better than more popular platforms at the time, for example the PHP and Perl-based CMS's), but manageable. Ever since 4.x it has become completely unmanageable unfortunately, and just getting worse. Hopefully we can figure this out. I'm still trying to get the buildout to actually upgrade the recaptcha version. Hopefully I will have that working soon, then will see if any difference with v2.
It doesn't really have any difference with Plone 4. The difference is the bots are much more sophisticated now. An email verification BEFORE any use creation might be the best route.
OOTB Plone is not usable with self registration right now though it seems. On Jun 16, 2015 6:55 PM, "merpdotcom" notifications@github.com wrote:
I've been hosting self-registration for dozens of community sites since the late 90s. In 2003/4 I switched them over to Zope-Plone. It had some challenges with registration spam, forum spam, contact spam (but was much better than more popular platforms at the time, for example the PHP and Perl-based CMS's), but manageable. Ever since 4.x it has become completely unmanageable unfortunately, and just getting worse. Hopefully we can figure this out. I'm still trying to get the buildout to actually upgrade the recaptcha version. Hopefully I will have that working soon, then will see if any difference with v2.
— Reply to this email directly or view it on GitHub https://github.com/plone/Products.CMFPlone/issues/632#issuecomment-112605852 .
That does unfortunately seem to be the case. :-(
@merpdotcom Can you try installing this package: https://pypi.python.org/pypi/collective.emailconfirmationregistration
It's rather simple but if it works out well, it's a concept I'd like to consider merging into Plone core.
:+1: we, at freitag.de, use this approach (first confirm the email, then register) and so far works perfectly.
First confirming before registering seems a much better route (thought that was what the core feature was supposed to anyway, rather than the temporary accounts piling up). @vangheen I will try that too.
@vangheem , what method should I use to install it? I tried as standard buildout and it didn't find, then tried adding as github source. Do I need to do something specific to add pypi-related products? My apologies for not having that down, appreciate directions for installation of the emailconfirmationregistration product. Thanks.
we're using quintagroup.formlib.captcha
on a plone4.3 site with self registration. there are spam registrations, but only a few (about 10 per week or month).
if you activated member folders, you can also use the following script to clean up spam users. it checks for users w/o a member folder and deletes them (it does not delete users with an active password request):
def _removeSpamUsers(portal):
pm = portal.portal_membership
pwr = portal.portal_password_reset
mdtool = portal.portal_memberdata
acl_users = pm.acl_users
ids = pm.listMemberIds()
logger.info("Checking %d users for spam-accounts (w/o a userfolder)" % len(ids))
#userids that requested a password reset
#{'aaron': DateTime(xxx), 'seth': DateTime(...),...}
pwd_requests_by_user = dict((user, expires) for user,expires in pwr._requests.values())
#all available member folders
member_folder_ids = pm.getMembersFolder().objectIds()
delCount = 0
for memberId in ids:
if not memberId in member_folder_ids:
if memberId in pwd_requests_by_user:
# skip members that just registered and are about to reset their passwords
logger.info("Skip %s with password reset request (expires %s)" % (memberId, pwd_requests_by_user[memberId]))
continue
member = pm.getMemberById(memberId)
name, email = (member.getProperty('fullname'), member.getProperty('email'))
logger.info("Delete %s - %s (%s)" % (memberId, name, email))
mdtool.deleteMemberData(memberId)
pm.deleteLocalRoles(portal, [memberId], reindex=1, recursive=0 ) #speedup: recursive not needed, member does not have any content
acl_users.userFolderDelUsers([memberId,])
delCount += 1
logger.info("Deleted %d spam members out of %s initial members" % (delCount, len(ids)) )
@vangheem It is installed now. Is there anything to configure or should it just work now?
As far as I can tell, it is still creating the account first. Maybe I'm not understanding what to check for you. Please clarify what I should be looking for so I can assist and testing it out. Thanks. Meanwhile I did finally get recaptcha v2 working correctly.
@vangheem, this is the test server registration page:http://www.spokanerpg.com/@@register feel free to hit it as needed. I deleted all created accounts except my admin-level account (there were three test accounts I had created, plus two others I didn't know from api.remailed.com (maybe someone helping test?). So slate is clean, any accounts created are either you/us testing, or something else.
Here is how the buildout.cfg looks now that recaptchan v2 is working, but I also tried your product. Buildout ran it. And I was able to enable it through site-setup products add-ons web page.
Thanks for trying to help to figure this nightmare out, I am going to have to call it a night for now. (5:31 am here).
Recaptcha v2 at least it appears to be working now, had to use the github grab (the pypi one has a zcml error they (jensen in IRC said) are fixing).
I don't think the emailconfirmation product is working yet, when I run a test, it creates the account as soon as I complete the form/recaptcha.
I do receive the activate email with the link. But even without responding/clicking it, the test accounts are now listed.
Meanwhile will see if the spam registration bots still get past the recaptcha v2.
I haven't slept in 93 hours trying to get this resolved, so will hit the hay for a while, then see if anything happened while I slept. And I will check back here for any directions to follow for testing your emailconfirmation product.
Thanks for all the help.
If this is finally fixed after a year of over 20+ previously vibrant community sites having user registration disabled, it will be such a relief to get the communities alive again.
I will let those that helped know how it goes. It might take a few days to a week to be sure (the bots sometimes don't attack right away). thanks again. ciao.
Sorry, I went to sleep myself.
Looks like collective.emailconfirmationregistration and collective.registrationcaptcha both customized the register form and collective.registrationcaptcha version was being used.
Please upgrade to the latest version of https://pypi.python.org/pypi/collective.emailconfirmationregistration/1.0.0a3 that I just released that should fix it. Let us know if you need help/instructions on installing a specific version of a package with buildout(look at your versions section in buildout.cfg).
Ok, I've tried pinning the version. I deleted all previous src files, but it is still grabbing a1 for some reason. Here is grepped buildout -vvv output, and the buildout.cfg You see what I am doing wrong? Thanks for guidance. http://pastie.org/10246074
@merpdotcom couple things here:
You don't want to run from master in github, you want to download the released version 1.0.0a3.
@vangheem thank you, that did the trick. 1.0.0a3 now in place and testing... tried for valid email address (only field offered), after a bit of a wait, "Warning: Could not verify email address you have provided ". I do not see anything in client1/event.log nor zeoserver.log. Suggestions?
hhmm, I'm using https://pypi.python.org/pypi/validate_email to validate the email address with the verify=True to check to make sure the email really exists.
I can just remove that check. I thought it'd be good though--worked in my tests. If there was a long wait, perhaps the server isn't allowing the outbound connection for checking from the python process?
@merpdotcom fwiw, I made a 1.0.0a4 release that disables that check. Maybe give that a try.
@vangheem Ok, upgraded to a4. Email confirmation sent. Received in Inbox. Also verified did not create any kind of account (can't imagine it could at this stage), so still no new accounts (good).
Email: You have requested registration, please confirm your email address by clicking on this link.
If that does not work, copy and paste this urls into your web browser: http://www.spokanerpg.com/@@register?confirmed_email=merpdotcom@gmail.com&confirmed_code=a67ff23c4d6f51b725b10ab25846f511e387be79
Clicked link: We’re sorry, but there seems to be an error…
The error has been logged as entry number 1434596062.910.682308390256.
Error from client1/event.log:
2015-06-17T19:54:22 ERROR Zope.SiteErrorLog 1434596062.910.682308390256 http://www.spokanerpg.com/@@register Traceback (innermost last): Module ZPublisher.Publish, line 138, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 48, in call_object Module collective.emailconfirmationregistration.browser, line 202, in call Module z3c.form.form, line 218, in call Module plone.z3cform.fieldsets.extensible, line 58, in update Module plone.app.users.browser.register, line 601, in updateFields Module plone.app.users.browser.register, line 116, in updateFields Module plone.autoform.form, line 34, in updateFields Module plone.z3cform.fieldsets.extensible, line 64, in updateFields Module collective.registrationcaptcha.registrationform, line 86, in update Module plone.z3cform.fieldsets.extensible, line 36, in add Module plone.z3cform.fieldsets.utils, line 28, in add Module z3c.form.util, line 298, in add Module z3c.form.field, line 136, in init ValueError: ('Duplicate name', 'captcha')
Tried copy and paste of the link provided for copy and paste:
2015-06-17T19:57:33 ERROR Zope.SiteErrorLog 1434596253.430.619347658767 http://www.spokanerpg.com/@@register Traceback (innermost last): Module ZPublisher.Publish, line 138, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 48, in call_object Module collective.emailconfirmationregistration.browser, line 202, in call Module z3c.form.form, line 218, in call Module plone.z3cform.fieldsets.extensible, line 58, in update Module plone.app.users.browser.register, line 601, in updateFields Module plone.app.users.browser.register, line 116, in updateFields Module plone.autoform.form, line 34, in updateFields Module plone.z3cform.fieldsets.extensible, line 64, in updateFields Module collective.registrationcaptcha.registrationform, line 86, in update Module plone.z3cform.fieldsets.extensible, line 36, in add Module plone.z3cform.fieldsets.utils, line 28, in add Module z3c.form.util, line 298, in add Module z3c.form.field, line 136, in init ValueError: ('Duplicate name', 'captcha')
For testing right now it is fine without recaptcha, but for production, would be important to have at least recaptcha (optionally both recaptcha with norobots option on same page as double check, would be ideal), so that spammers can't use the form for backscatter attacks, or just overloading my poor mail server. :-) But we can worry about those features later. Happy to test away!
sigh, sorry about that. Like I said, I didn't test much. Just made another release to fix 1.0.0a5--give it a try please.
As for the validate feature in a3, the test server is running from my home over a Comcast cable residential connection. I am testing using a valid gmail account. It is possible since it is coming from a dynamic IP from a residential connection that added delays or checks are interfering. I will be moving this server to the colo facility this Friday. It will become production for the sites (I have to do this, older server is overloaded), but I can setup a separte test instance to test on the colo from time to time. I will be wiping the old server (once everything transfered), and it will be setup identically as the new prod server, so I can use the dev server from home for staging. So most testing will still be from home, but there will be this transitional period of several days (weeks?). When the changes happen, if we are still working on this, I will specific with each test result whether it was on home-dev server or colo-prod or colo-dev instance. Okay?
No worries about the errors, each iteration is making progress. I am happy to be your dev code tester. :-) (retired CTO/CIO)
Testing a5. Buildout ran fine. Sent email. Email received. You have requested registration, please confirm your email address by clicking on this link.
If that does not work, copy and paste this urls into your web browser: http://www.spokanerpg.com/@@register?confirmed_email=merpdotcom@gmail.com&confirmed_code=e452a6061b883ef58fae3b87f5a5535eacc03075
Clicked first link, presented with full registration form now, with Recaptcha. Verified no other accounts in users listings. Completed registration form (noticed I can put any other email address I want in there now, rather than the verified email): I put in a different email address than the verified email (just to see what happens). Completed REcaptcha clicks. Clicked Register Generated error: Browser URL now showing: http://www.spokanerpg.com/@@register But still standard form rather than just short email form. Prompted to correct error with email address: tried with untested email again (just to see), Error prompt again to correct e-mail address Tried it this time with verified email address. Received confirmation page:
Welcome! You have been registered.
You will receive an e-mail shortly containing a URL that will allow you to set your password. When you receive this e-mail, please follow the link to complete your registration process. NOTE: The password reset URL will expire on Jun 24, 2015 08:12 PM. If your password reset request expires, you may use the "Forgot your password" link to generate another password reset request. Account now shows up in Users list.
Excellent!
Received user account creation email with link to "Activate" account Welcome test, Your user account has been created. Your username is test. Please activate it by visiting http://www.spokanerpg.com/passwordreset/278ee8c605d3f06d41b20feed48b8c49?userid=test Please activate your account before Jun 24, 2015 08:12 PM
(Is it possible to change the duration that is valid to something more like 24-48 hours?)
Prompted by passwordreset page: completed password (twice), and clicked submit: +++++++++++++++++ ERROR Error setting password Sorry, this appears to be an invalid request. Please make sure you copied the URL exactly as it appears in your email and that you entered your user name correctly. ++++++++++++++++++
NOTE: Did not generate error in client1/event.log
Well, we're getting closer by the moment! :-)
Actually, this might be user error. I think my web browser auto-completed a different username than test, and I think your code caught it, but didn't have any exception handling to point out it was a different user name. WIll test that in a second.
But here is with following the desired use-case with no attempts to enter wrong email or username:
I can see some users being annoyed by all these steps (two emails to set account and then set password), but hopefully in this day and age users will be more understanding about how problematic it is to fight the spammers.
Tested again, just cooperating and using the valid email the entire process instead of trying to "trick" it, Worked through whole process, including password without error. It states on next page: "Info: Verification email has been sent to your email. " However I do not receive that confirmation email.
Okay, verifying if that password creation page error was from entering a different username than previously validated: All steps worked flawlessly. Account fully created. Logged in with it, no problems. I think the earlier error was user error, but it could use some exception handling to point out to the user they entered the wrong username, that it didn't match their previous validation.
I then went into my Site Setup > Sercurity, and enabled "Let users select their own passwords", to make life easier for users, since at this point we've gone through email validation, and then recaptcha, it might be robust enough to get away with it? Tested, and everything worked fine as well! The Login button was right there conveniently for the user, clicked it, and it immediately logged in without having to re-enter the username and password I just entered a moment before. Not sure if bots can still abuse that or not, but that is definitely preferred user experience. but at least now there is the additional step as an option.
EXCELLENT!
(Is it possible to change the duration that is valid to something more like 24-48 hours?) unfortunately, I don't know how to edit that valid without a patch. I could be wrong though.
Yes, your comments about there being too many steps are justified. Long term, this form needs to be more streamlined. I think, for now, it's important we get filtering out the bots the best we can though.
Thank you for testing. Please update us with info as far as if bots are able to automate the email verification steps. I think there are a few more tricks we can try. For instance, even doing the honey pot stuff is really easy to implement and might help. Unfortunate the email dns lookup didn't work. I wish there was an open web server api or something for checking an email against know bad domains/email addresses or something. Annoying problems spammers are these days...
@vangheem This is an excellent start, and well worth it! I will definitely let you know if any baddies get through! :-) Thank you so much for helping me through this. Maybe I can revive some of the dead communities now.
There are RBLS blacklists, graylists, and white lists versus IP addresses (I use them for my mail servers constantly). As for email addresses, there wouldn't be any point, they can autogenerate any bogus email they want, so it would seem to be too much overhead with too little potential reward. At least with the option for IP-based blocking, known attack servers or infected users, could be blocked, But you are getting into a whole morass of technologies there. I think what you did here is a much cleaner approach. And if it works to stop them fully, the Plone folks should definitely consider adding it as a checkbox or standard product to bundle to help many others. :-)
Cheers!
@vangheem Oh, though if possible, it would be very good to have the Recaptcha option on the first step email form entry, otherwise backscatter will be a serious problem. Is that trivial for you to add, or more complex?
Sure, it'd be easy to move it there. Then we'd just remove the use of that collective.captcharegistration package(assuming you don't want to fill out 2 recaptchas). I'll try putting an update out tomorrow morning. On Jun 17, 2015 10:53 PM, "merpdotcom" notifications@github.com wrote:
Oh, though it would be good to have the Recaptcha option on the first step email form entry, otherwise backscatter will be a serious problem. Is that trivial for you to add?
— Reply to this email directly or view it on GitHub https://github.com/plone/Products.CMFPlone/issues/632#issuecomment-113027694 .
That would be great. Then fewer inter-package dependencies, and putting the recaptcha at the first layer (email). Seems cleaner. Thanks again!
@vangheem Rest well. The good news is that so far, though the bots are trying already, the the email validation is preventing them from creating bogus accounts. The down side, as was expected, is they are starting to generate backscatter. Registration validation emails are already being sent from the server about one per minute right now. In the past this typically can ramp up to many per second. Hopefully the Recaptcha v2 on the email validation will stop them in their tracks when it is put in place. Just thought I'd let you know the traffic attempts had begun already, that at least helps with testing. And this is not one of my normally busy sites. :-) Thanks again!
@vangheem FYI, just finished upgrade to 4.3.6 (it was kind of messy, but eventually made it there). Fortunately your code appears to work exactly as well as before. Just thought you would want to know it has now been tested on both versions.
Alright, give version 1.1.0b2 a try. This moved the recaptcha field to the email verification page.
Additionally, it adds a honey pot hidden username field.
@vangheem Updated to b2. Verified working as desired. Did not perform other use-case testing, just optimal use-case compliance, but worked perfectly doing that. This is running 4.3.6. Will test attempts to "break" it, and if anything noted will let you know. Will keep you posted if any bots figure out how to bypass. This server will be going into prod tomorrow, with 26 different community sites, so that will be when the heavy testing will be. The dev server is hosting 3 live community sites now with this code, and the logs show the bots going mad trying to get in, but so far they are zero. Thanks again!
@merpdotcom great! Thanks.
@tisto @bloodbare @frapell others? Does it make sense to try and squeeze this into plone 5?
I would definitely vote for it! :-) It looks like the registration spam blocking techniques (may) have thwarted over 700 attempts so far today!
This might be what you want down the road @vangheem http://botscout.com/ ?
@vangheem interesting feature/bug? Everything works correctly if creating user through normal means. However if I create a user account as admin (probably any manager), the account is created just fine, but when that user then logs in, they are prompted with the confirm email page: Confirm your email address Before you can begin the registration process, you need to verify your email address. I can ignore it and go to other pages. The account works normally. Thought you would want to know. Also, the bots are succeeding in bypassing the recaptcha v2, I am getting occasional backscatter bounced emails for the initial stage 1 very email process. It is not very many (out of thousands of attempts to far, I have seen 3 get through). Possible to include both recaptcha and norobots on that first stage easily enough? However, still ZERO bogus accounts created across all sites so far, and the bots are trying thousands of times per day. Cheers!
@merpdotcom thanks for your help on this. I am closing the issue now.
For now, we'll advise people to install collective.emailconfirmationregistration on Plone 4.
I'll see if it is reasonable to merge this into plone 5 during the anniversary sprint; otherwise, this will be PLIP'd for plone 5.1.
Bad news, it appears "they" have figured out how to bypass on at least one of my sites now. :-( There are now over 140 fake registrations on the tolkienmoot website that had been working fine to stop them. This is version 1.1.0.b2. Please advise on what I should do to get you information on how they are once again bypassing? I will check the other sites to see if any others are showing registration spam, or if it is only one site.
Some additional sites indeed have been bypassed, brencode, hawketalkshow, omzine, rpgresearch (over 500+!), tolkienscholars, but not merpdotcom, spokanerpg, or half a dozen other sites. The rest (~20 I still hadn't enabled self-registration pending the results of this testing). I am disabling self-registration once again for the "compromised" sites until we can figure out what to try next. Thanks in advance for any suggestions you may have.
Does this issue need reopening?
Version 1.20.b2 has now been created to add the option for an admin to verify/authorize a registration request as an additional layer to try to combat this problem. b2 has a bug that doesn't send an email if "let use select own password" is checked, that should hopefully be fixed soon. I am now testing b2 on the several sites that were still getting bypassed to see if that works. https://pypi.python.org/pypi/collective.emailconfirmationregistration
Crud, it looks like somehow it is still being bypassed. Accounts are being created without ever going to the admin for approval. :-( 17 accounts since yesterday.
I don't think this issue needs reopening. We are dealing with issues on the add-on package tracker. We'll target 5.1 to move add-on into plone.
The hotfix worked beautifully for a a while. But now it looks like they have figured out how to bypass it again. :-( Now about 13 new bogus accounts in about 2 days. Nothing in the "Review Registration Requests". Have they figured out another Plone vulnerability? http://techtalk.hawkenterprising.com/news/important-plone-hotfix-20150910
@merpdotcom maybe in future if you suspect a vulnerability could you email the security team directly rather than post here (a public place)? security @ plone.org
Registration spam has been completely out of control for over a year now. I have had to disable user self-registration on all sites. I kept being told to upgrade to newer Plone, and to use captcha/recaptcha. A long process with 20+ previously active community sites. Finally managed to setup fresh server with 4.3.5 (on linux).
I added collective.registrationcaptcha and collective.z3cforms.norobots. Configured no robots with extremely esoteric and oddball questions.
Example url: http://www.spokanerpg.com/@@register
I enabled self-registration, but disable "let use select their own passwords", enable user folders. When I test registration page, it shows the correct norobots prompt. It will not let me register until correctly answered. But I am getting horrible registration spam, 20-50+ per site per day on just the test server (thousands on the main server if registration enabled). This is no different than on the earlier 4.2 and 4.1 boxes that were trying to use captcha, recaptcha, etc. WIthin a few days there are thousands of bogus accounts, and the mail server begins to get loaded from all the registration email processing (many of which bounce as bogus emails).
Is plone registration being injection bypassed or some such?
Please help, this has completely crippled more than 20 different community websites that used to be active and vibrant.
Thank you.
Sample Email bounce from bogus registration (get hundreds of these per day, sometimes per hour:
Here is sample bounced mail to a bogus email account from a bogus registration:
From: Mail Delivery Subsystem mailer-daemon@… To: spokanerpg@… Subject: Delivery Status Notification (Failure) Date: Tue, 16 Jun 2015 09:29:29 +0000
Delivery to the following recipient failed permanently:
Technical details of permanent failure: Google tried to deliver your message, but it was rejected by the server for the recipient domain mail.bg by mx-balancer.mail.bg. [2001:67c:16b8::ffff:2].
The error that the other server returned was: 550 5.2.1 erraticdepictio35@…: Recipient address rejected: Mailbox is temporarily on hold, not accepting messages
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
X-Received: by 10.70.127.140 with SMTP id ng12mr57235605pdb.94.1434446967093;
Return-Path: spokanerpg@… Received: from d2d203.dev2dev.com (c-73-221-115-191.hsd1.wa.comcast.net. [73.221.115.191])
Message-ID: 557fec76.c9f5420a.7198.176f@… To: erraticdepictio35@… Precedence: bulk MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Subject: =?utf-8?q?User_Account_Information_for_Spokane_RPG_Role-Playing_Gaming_Co?=
From: "SpokaneRPG.com Webmaster" spokanerpg@… Date: Tue, 16 Jun 2015 02:29:24 -0700
-- SpokaneRPG.com Webmaster