major spam attack by a bot

[tfoote]

They're creating new users and posting one question per user.

It also seems that the questions for new users can link and it doesn't have the nofollow attribute.

[tfoote]

We've turned off password based registration for the moment.

[trainman419]

I'm seeing this again today

[tfoote]

There were some 50-100 spam posts again. From 4-5 users.

[evgenyfadeev]

Hi Tully,

I'm thinking what would be the most effective method. Perhaps user moderation? For example a user with a "watched" status is allowed to make one post after which we'd have to decide whether to allow posting anything else or not.

There is also Captcha, but supposedly the the ReCaptcha has been solved and can be overcome with automation.

Other possibility - limit account creation per IP address. It may be too restrictive though.

The issue is that no matter what, the users will be able to post something manually. So I think we'd have to go the "moderation" route.

Please let me know what you think.

Thank you, Evgeny.

On Tue, May 27, 2014 at 1:21 AM, Tully Foote notifications@github.comwrote:

There were some 50-100 spam posts again. From 4-5 users.

— Reply to this email directly or view it on GitHubhttps://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-44234668 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[evgenyfadeev]

I will give captcha a spin, but it's possible we'll need a set of features to reduce spam.

On Tue, May 27, 2014 at 1:39 AM, Evgeny Fadeev evgeny.fadeev@gmail.comwrote:

Hi Tully,

I'm thinking what would be the most effective method. Perhaps user moderation? For example a user with a "watched" status is allowed to make one post after which we'd have to decide whether to allow posting anything else or not.

There is also Captcha, but supposedly the the ReCaptcha has been solved and can be overcome with automation.

Other possibility - limit account creation per IP address. It may be too restrictive though.

The issue is that no matter what, the users will be able to post something manually. So I think we'd have to go the "moderation" route.

Please let me know what you think.

Thank you, Evgeny.

On Tue, May 27, 2014 at 1:21 AM, Tully Foote notifications@github.comwrote:

There were some 50-100 spam posts again. From 4-5 users.

— Reply to this email directly or view it on GitHubhttps://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-44234668 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

Askbot Valparaiso, Chile skype: evgeny-fadeev

[tfoote]

​Lets give it a try. If we can make it just a little harder for the spammers they might go elsewhere.

[skohlbr]

It appears the spam attacks in recent days are all in arabic and targetting "customers" in egypt. Would it make sense to install filters that prevent posting of messages using majorly non-english alphabets?

[BennyRe]

According to this changelog askbot supports Akismet.

Could this help to prevent the spam?

[dlaz]

Is there any sort of rate limiting for posting questions? Limiting single users to only posting a few questions per hour could at least make moderation a little more manageable for the community.

[evgenyfadeev]

There isn't, but actually rate limiting is not a problem to bypass, you can just schedule spam posts by script. Rate limiting can help in some cases.

I'm almost done with recaptcha and implementing "watched" user status, where "watched" users will be forced to use recaptcha on every post (also on registration).

On Wed, May 28, 2014 at 12:53 PM, Dan Lazewatsky notifications@github.comwrote:

Is there any sort of rate limiting for posting questions? Limiting single users to only posting a few questions per hour could at least make moderation a little more manageable for the community.

— Reply to this email directly or view it on GitHubhttps://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-44426305 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[tfoote]

Recaptcha for posting sounds good. And "watched" means that they're below X karma? The spammers seem to come back almost every night and can get through the open ID registration requirement.

[evgenyfadeev]

Watched atm means - uses captcha, also there will be a threshold for auto-approving user (change status from "watched" to "approved") when their reputation crosses the margin from below.

So it will be possible to force users to use captcha.

On Wed, May 28, 2014 at 2:47 PM, Tully Foote notifications@github.comwrote:

Recaptcha for posting sounds good. And "watched" means that they're below X karma? The spammers seem to come back almost every night and can get through the open ID registration requirement.

— Reply to this email directly or view it on GitHubhttps://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-44441124 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[evgenyfadeev]

Enabled captcha and made all users with karma > 10 bypass captcha. Users with "watched" status and anonymous visitors will have to pass the captcha test:

• when registering
• asking, answering, editing question or answer
• when sending feedback

Users who reach karma 10 (adjustable) will become auto-approved.

If you want to force users to use captcha - change their status to "watched".

Let's see how this works.

[evgenyfadeev]

sorry, disabled for now, will update shortly.

[evgenyfadeev]

Should work now. We should now see questions and answers posted by new and low rep users (<10). Users should also be able to register and edit posts.

It would be best to test this on a live site, maybe someone could try creating a new account and make some reasonable questions/answers and edit them?

Works on my development computer. On answers.ros.org I've tested feedback form and registration.

[tfoote]

We're still getting 2-3 attacks per day. And they're posting 3 to 20 posts each.

I tried out using the captcha, and it's a little awkward when you fail it that it reloads the page and there's no indication to retry, and doing it from an answer section the answer is folded up with your content missing.

And I did not seem to be able to register without an OpenID yet. I guess that's not enabled yet? It's hard to test the registration by creating more openID accounts.

[tfoote]

I did create a test account, and was able to post a question. I was not able to post a question with an external link. (However it did not give any feedback as to why it would not post. Whereas if I failed the captcha it told me to try again in red.)

[tfoote]

It did require the captcha for registering and posting. I think we may be up against a real person. I'm not sure what else we can do. Though the useability comments above would be nice to fix.

[evgenyfadeev]

The immediate plan is to implement pre-moderation of new users posts, tools for mass clean-up and the IP blocking.

Best regards, Evgeny.

On Tue, Jun 10, 2014 at 4:17 PM, Tully Foote notifications@github.com wrote:

It did require the captcha for registering and posting. I think we may be up against a real person. I'm not sure what else we can do. Though the useability comments above would be nice to fix.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-45658894 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[tfoote]

I'd strongly prefer to make the cleanup easier. Moderating all new users will greatly decrease the initial impression on the users and add more overhead and another stream to require everyone to monitor.

If we could make it such that it was just a few clicks to ban a user and delete all their posts that would be fine. Right now when I get 30 incoming posts. I need to click on each one to clear it from the review queue, and I also have to check that the user has been blocked. And then check that all their posts have been cleared. Adding this sort of thing to the drop down on the review page would be great.

If the review queue could make the user status and post status more visible such that if another admin has already deleted and blocked the user I could know that and just mark as read that would be very helpful.

On Tue, Jun 10, 2014 at 12:23 PM, Evgeny Fadeev notifications@github.com wrote:

The immediate plan is to implement pre-moderation of new users posts, tools for mass clean-up and the IP blocking.

Best regards, Evgeny.

On Tue, Jun 10, 2014 at 4:17 PM, Tully Foote notifications@github.com wrote:

It did require the captcha for registering and posting. I think we may be up against a real person. I'm not sure what else we can do. Though the useability comments above would be nice to fix.

— Reply to this email directly or view it on GitHub < https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-45658894>

.

Askbot Valparaiso, Chile skype: evgeny-fadeev

—

Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-45659570 .

[gavanderhoorn]

@tfoote wrote:

I think we may be up against a real person. I'm not sure what else we can do.

If we are they are persistent: just removed 20+ posts by three different accounts (yasooo2014 and saaraaa. Forgot the 3rd one).

[skohlbr]

What we´ve seen in recent weeks was mainly arabic spam, so just mentioning again my idea of preventing posting of (majorly) non-english alphabet postings. Not sure about technical feasibility, but should prevent the current style of spam attacks (or force them to switch to english ;) ).

[evgenyfadeev]

Deployed feature - "block user and delete all content" in "profile->moderation". Please try this.

[tfoote]

Great! It's there and seems to be working.

I think the last useability issue is that moderation/flag review area doesn't work well with multiple users. If I go to review my queue and someone else has already cleaned up I have no way to know that the flag has been dealt with by another moderator.

What I'd love to see would be a single queue with individual new vs viewed, and a history next to each item for "flagged by X (+ Y + Z)( deleted by moderator W | approved by moderator V)

And preferably the moderation result could also be overturned from that view keeping the history of actions on the post. One of the issues I've had is if a post is accidentally moderated/deleted. It dissapears from the review queue, and then you need to remember the user profile to undelete the question or answer. (Since you can find it on their user profile still)

[tfoote]

Ohh, it would also be great to see the user Moderation Status, aka "User Blocked" on the review page too.

[tfoote]

Just randomly I spot checked one of the spam numbers and it appears to have gotten onto google. The link google points to is invalid now when I browse, not logged in as an admin. However are we sure Google's not finding it still?

It appears that whoever is doing this is doing it all over the internet from the search too. There's posts on youtube, vimeo, adobe forums, facebook...

[evgenyfadeev]

Maybe we automatically add blocked content into the robots.txt?

On Sun, Jun 22, 2014 at 2:09 AM, Tully Foote notifications@github.com wrote:

Just randomly I spot checked one of the spam numbers and it appears to have gotten onto google. [image: spam_on_google] https://cloud.githubusercontent.com/assets/447804/3350679/923eef64-f9c9-11e3-8d6a-6f39bb875fbf.png The link google points to http://answers.ros.org/question/173167/lw-yz-trf-rqm-twkyl-bwsh-blskndry01227713067tsl-ln/ is invalid now when I browse, not logged in as an admin. However are we sure Google's not finding it still?

It appears that whoever is doing this is doing it all over the internet from the search too. There's posts on youtube, vimeo, adobe forums, facebook...

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-46772714 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[tfoote]

Looking a little close it appears that it's just google being really quick to index the site, in the 4-8 hours it takes for us to clean up the spam.

The new block and delete button is great! I just cleaned up ~ 100 posts from overnight in less than 5 minutes.

From the recent behavior I think flood control might be reasonable. One of the posters posted 68 questions overnight. If we make it 1 question every 5 minutes max or something like that it would make the mass postings much harder. Or we could allow multiple questions but require a longer backoff. Maybe 3 per hour sort of thing while a watched user.

Other forum spam fighting techniques are listed at: https://en.wikipedia.org/wiki/Forum_spam ​

[dornhege]

A huge +1 for "block user and delete all content".

I guess now it's moderate and see? That bot seems quite resilient.

[gavanderhoorn]

On 23-6-2014 13:40, Christian Dornhege wrote:

A huge +1 for "block user and delete all content".

I'm guessing this is only for moderators?

[NikolausDemmel]

On 23-6-2014 13:40, Christian Dornhege wrote: A huge +1 for "block user and delete all content". I'm guessing this is only for moderators?

I guess so, I don't see this with 1000+ karma. Spam is getting out of hand, almost every time I check the website it is mostly Spam posts. Especially when it is night time in California.

Now we also get English SPAM and from lots of different account at the same time as well.

Limiting posts to 5/day or something for watched users sounds like a good idea.

I didn't quite get if there is a catcha now for registration, or not?

[NikolausDemmel]

Also, IMHO relying on some external service like Aksimet sounds like a good idea.

[gavanderhoorn]

On 29-6-2014 12:14, Nikolaus Demmel wrote:

On 23-6-2014 13:40, Christian Dornhege wrote: A huge +1 for "block user and delete all content". I'm guessing this is only for moderators?

I guess so, I don't see this with 1000+ karma. Spam is getting out of hand, almost every time I check the website it is mostly Spam posts. Especially when it is night time in California.

I must admit I'm completely ignorant of the current setup, but are there moderators in the different time zones?

[dornhege]

This is getting out of hand. I just deleted multiple user accounts with a total of more than 1000 questions and new accounts are coming in by the minute. Now instead of deleting dozens of questions it's manually deleting dozens of accounts manually.

Rate limiting is reasonable - I never had more than 1 question/day and I can't imagine someone needing more than 3.

Still, the bot is bypassing any registration hurdles there are. Unless there is a proper solution to that right now I think the only way is moderating new user questions (at least the first one). This won't deter new users any more than a site full of spam.

[tfoote]

Yeah, I did the same and so did @mikeferguson

I also just got some alerts for an answer which was spam so they're exploring more options too.

[evgenyfadeev]

I'm working exactly on this feature now - the pre moderation.

On Sun, Jun 29, 2014 at 1:39 PM, Tully Foote notifications@github.com wrote:

Yeah, I did the same and so did @mikeferguson https://github.com/mikeferguson

I also just got some alerts for an answer which was spam so they're exploring more options too.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-47459772 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[130s]

Meanwhile I usually flag offensive and delete posts (and hit the daily max (20 for me) of doing so). Is it the best that normal registered users can do?

[evgenyfadeev]

Isaac, as a normal user, this is all you can do. Actually don't bother flagging - just delete the posts.

The moderators can also delete content of users in bulk and block users.

On Sun, Jun 29, 2014 at 9:10 PM, Isaac Isao Saito notifications@github.com wrote:

Meanwhile I usually flag offensive and delete posts (and hit the daily max (20 for me) of doing so). Is it the best that normal registered users can do?

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-47485184 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[mehditlili]

There is a new attack right now from some Indian astrologers. Isn't there a feature that permits users to flag other users and not only their questions? A user flagged by 5 or more other users could be put on hold until a moderator checks what is going on.

[ccapriotti]

Domhege suggested I posted here; people, if you needs hands on deck to moderate those messages, I am volunteering, supposing the system allows for it.

[dlaz]

So far, it doesn't seem like anything has been able to proactively stop these spam attacks. That said, humans in the loop is probably our best bet, and there are clearly lots of community members willing to help. I wonder if imposing some automatic consequences for users with flagged posts would help. For example, a user with 3 flagged questions in the past day is rate limited, or has to fill out a captcha for any new questions. This can at least keep things manageable until an admin has a chance to make a decision.

[ccapriotti]

IMH, captcha for registration immediately, along with close monitoring of new members is the way to go.

Now, something else is also concerning me: deleting all the crap from the history on the web. But that is for later, of course.

Again, count on me.

An idea for monitoring: is there a way to create a special level of mediation, to be given to a "lot of people" with the only objective of stopping this (type of) attack ?

Infantry... We take the bullet for you guys, and leave only the serious stuff, pre-filtered, for the system or you guys. Task-force-like, this could be a team that is called to action whenever there is an outburst.

Food for thought.

On Mon, Jun 30, 2014 at 5:06 PM, Dan Lazewatsky notifications@github.com wrote:

So far, it doesn't seem like anything has been able to proactively stop these spam attacks. That said, humans in the loop is probably our best bet, and there are clearly lots of community members willing to help. I wonder if imposing some automatic consequences for users with flagged posts would help. For example, a user with 3 flagged questions in the past day is rate limited, or has to fill out a captcha for any new questions. This can at least keep things manageable until an admin has a chance to make a decision.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-47543062 .

[NikolausDemmel]

So far, it doesn't seem like anything has been able to proactively stop these spam attacks. That said, humans in the loop is probably our best bet

Am I wrong or is the only thing that has been tried captcha for registration? Aren't there lots of tools to try to detect spam automatically, e.g. Akismet?

If there are lack of resources to implement this right now, moderating new users should be done (temporarily) as immediate action.

[dlaz]

This is getting out of hand. The first 3.5 pages are currently spam, and I'm sure Tully can't devote all his time to dealing with it. There needs to be a short term fix until something is worked out - temporarily disable user registration, make a few more admins, etc.

[mikeferguson]

I just cleaned up the existing spam. Perhaps, for this particular spammer, a quick check of "the word marriage exists in this post" would suffice to auto block them...

[ccapriotti]

People, if this is open to a vote, I agree with suspending new subscriptions for a few days.

All necessary apologies to users and action.

ROS' image is at stake here. Not dealing with this properly may cause damage beyond repair.

Again, I am willing to help. On Jul 1, 2014 7:15 PM, "Michael Ferguson" notifications@github.com wrote:

I just cleaned up the existing spam. Perhaps, for this particular spammer, a quick check of "the word marriage exists in this post" would suffice to auto block them...

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-47684039 .

[evgenyfadeev]

Hi guys,

Please let me know if you want to implement the new users registration freeze.

It should take me two days maximum to finish pre-moderation.

Best regards, Evgeny.

On Tue, Jul 1, 2014 at 3:16 PM, ccapriotti notifications@github.com wrote:

People, if this is open to a vote, I agree with suspending new subscriptions for a few days.

All necessary apologies to users and action.

ROS' image is at stake here. Not dealing with this properly may cause damage beyond repair.

Again, I am willing to help. On Jul 1, 2014 7:15 PM, "Michael Ferguson" notifications@github.com wrote:

I just cleaned up the existing spam. Perhaps, for this particular spammer, a quick check of "the word marriage exists in this post" would suffice to auto block them...

— Reply to this email directly or view it on GitHub < https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-47684039>

.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-47691232 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[mikeferguson]

I don't think turning off new registrations if a very good idea -- in the last 24 hours, 6 people have signed up and asked legitimate questions. We've been dealing with this for several weeks now, two more days won't be that much worse.

[ccapriotti]

Michael, I respect your point, but honestly, I myself considered unsubscribing.

I came to my senses and simply switched off receiving emails from the list while this is still going on, but you can imagine the wide range of reactions this event is generating.

Six subscription in the last 24 hours is good, specially with valid questions, and that number tells me that moderating newcomers is feasible, if the load is distributed, preferably among people in different time zones around the globe.

On Tue, Jul 1, 2014 at 8:35 PM, Michael Ferguson notifications@github.com wrote:

I don't think turning off new registrations if a very good idea -- in the last 24 hours, 6 people have signed up and asked legitimate questions. We've been dealing with this for several weeks now, two more days won't be that much worse.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-47693389 .

[mikeferguson]

@ccapriotti So the part of your response that shocks me is: emails work for you? You get notifications? I was under the assumption that nobody was getting any emails anymore, because non of my subscriptions appear to work (and for a long time, none of them did for anybody). So yes, I could totally understand lots and lots of emails being annoying -- I was completely unaware that any subscriptions worked for anyone....