major spam attack by a bot

[tfoote]

They're creating new users and posting one question per user.

It also seems that the questions for new users can link and it doesn't have the nofollow attribute.

[ccapriotti]

Ah... never mind my last email.

I hadn't read Tully's post yet.

On Tue, Jul 1, 2014 at 10:46 PM, Carlos Capriotti < capriotti.carlos@gmail.com> wrote:

Michael, I respect your point, but honestly, I myself considered unsubscribing.

I came to my senses and simply switched off receiving emails from the list while this is still going on, but you can imagine the wide range of reactions this event is generating.

Six subscription in the last 24 hours is good, specially with valid questions, and that number tells me that moderating newcomers is feasible, if the load is distributed, preferably among people in different time zones around the globe.

On Tue, Jul 1, 2014 at 8:35 PM, Michael Ferguson <notifications@github.com

wrote:

I don't think turning off new registrations if a very good idea -- in the last 24 hours, 6 people have signed up and asked legitimate questions. We've been dealing with this for several weeks now, two more days won't be that much worse.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-47693389 .

[tfoote]

I've recruited several new moderators and it seems to have prevented the buildup of spam. @evgenyfadeev I don't think we need to freeze new user registration. I really hope that if the spammers will not find our site useful if their content does not stay. And with the more moderators approving first posts hopefully can be prompt as well.

@mikeferguson As far as i know subscriptions should be working. Please open a ticket if you're are not behaving as you expect.

[dornhege]

One quick note for new user's post moderation: It might be good to have an active trigger, e.g. an email to all moderators, so that questions will be activated as soon as possible. As that would only be for preventing spam it just applies to the first question of a new user and the traffic should be low.

[tfoote]

Even though we're being pretty prompt about deleting content. Google's apparently even faster at indexing it. This search shows 2890 results for "love" on ros.org https://www.google.com/search?q=site%3Aros.org+love

In looking into google's webmaster tools a related issue is the sitemap is invalid #84 We also talked about adding blocked urls to the robots.txt which might help too, but it appears they have a max size of 500k so we can't just add all the blocked pages. Hopefully with #84 will help.

[evgenyfadeev]

Fixed the sitemap issue.

On Wed, Jul 2, 2014 at 7:37 PM, Tully Foote notifications@github.com wrote:

Even though we're being pretty prompt about deleting content. Google's apparently even faster at indexing it. This search shows 2890 results for "love" on ros.org https://www.google.com/search?q=site%3Aros.org+love

In looking into google's webmaster tools a related issue is the sitemap is invalid #84 https://github.com/ros-infrastructure/answers.ros.org/issues/84 We also talked about adding blocked urls to the robots.txt which might help too, but it appears they have a max size of 500k so we can't just add all the blocked pages. Hopefully with #84 https://github.com/ros-infrastructure/answers.ros.org/issues/84 will help.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-47846344 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[skohlbr]

More spam coming in. I´m also available for moderation.

[BennyRe]

With my new moderator rights I just deleted 5 spam posts. But these posts haven't been posted by new users! The user accounts were real. One of them had over 1700 karma.

Do we have a security gap that allows hijacking user accounts?

[gavanderhoorn]

On 3-7-2014 9:10, BennyRe wrote:

With my new moderator rights I just deleted 5 spam posts. But these posts haven't been posted by new users! The user accounts were real. One of them had over 1700 karma.

Do we have a security gap that allows hijacking user accounts?

Are you sure that that account posted the spam, or did it just report (flag) the posts?

I find the moderation interface sometimes a bit unintuitive, and have had to look twice to make sure I wasn't blocking the user reporting the posts.

[evgenyfadeev]

If you have moderator rights, please go to the offending user's profile -> moderation and click "block and delete content".

On Thu, Jul 3, 2014 at 4:13 AM, G.A. vd. Hoorn notifications@github.com wrote:

On 3-7-2014 9:10, BennyRe wrote:

With my new moderator rights I just deleted 5 spam posts. But these posts haven't been posted by new users! The user accounts were real. One of them had over 1700 karma.

Do we have a security gap that allows hijacking user accounts?

Are you sure that that account posted the spam, or did it just report (flag) the posts?

I find the moderation interface sometimes a bit unintuitive, and have had to look twice to make sure I wasn't blocking the user reporting the posts.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-47874102 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[BennyRe]

Oh sorry.

I indeed thought the user that has reported the spam was the spammer, because of the user avatar.

[tfoote]

@BennyRe please make sure to unblock that user.

[ccapriotti]

How about raising the limit of "offensive flags" we mortals can use for a while ?

I exhausted my 20 in a single attack from our beloved guru.

On Thu, Jul 3, 2014 at 10:22 PM, Tully Foote notifications@github.com wrote:

@BennyRe https://github.com/BennyRe please make sure to unblock that user.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-47980237 .

[dornhege]

There should not be a need for that. It is sufficient to flag only one question per spam account. Moderators will block this user and delete all its content in one action.

[skohlbr]

Requesting moderator rights (my Karma should be sufficient for that :) ). There´s 4 pages of spam since more than an hour.

[trainman419]

This new user was able to create 310 posts before I blocked them: http://answers.ros.org/users/19480/oteiu/

They had "watched" status, but I don't think the current captcha is working properly, or it's too easy to defeat.

[dornhege]

I just blocked the first spammer from the moderation queue before it appeared on the site. +1 !

I'd like to have some more visual separation (hline, vspace, block, etc.) between entries in the moderation queue, so that it is clear which user/checkbox belongs to a post. I don't want to accidentally block a real user.

[NikolausDemmel]

At the moment in my moderation queue for "flagged items" it says there is 1 item, but it does not list any items.

Also, since recently in the "Forum responses" section the selection buttons ("all","none") as well as the checkboxes next to the items are missing such that I cannot discard items any longer.

[evgenyfadeev]

True, there is an issue of one item not being shown, I will have a look at that.

Regarding the "forum responses" - I've decided to change how this works. The "red alert" will be hanging for just some time after you look at the responses page and will automatically clear. There will be nothing to delete, to make it simpler.

Best regards, Evgeny.

On Wed, Jul 9, 2014 at 11:29 AM, Nikolaus Demmel notifications@github.com wrote:

At the moment in my moderation queue for "flagged items" it says there is 1 item, but it does not list any items.

Also, since recently in the "Forum responses" section the selection buttons ("all","none") as well as the checkboxes next to the items are missing such that I cannot discard items any longer.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-48478331 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[NikolausDemmel]

I'm not sure what you mean by "red alert"? I assume you are referring to the red envelope indicating new forum responses. AFAIR that alert used to always clear as soon as you looked at your responses. However I did like the ability to delete selective response items from the inbox to remind me where I haven't responded immediatly and would like to maybe do so later on.

Just to be clear, this is how my inbox looks: https://www.dropbox.com/s/vo306sk1fb1qr0a/Screenshot%202014-07-09%2016.40.23.png

[evgenyfadeev]

Yes, the red flag. I'll need to do some work to auto-clear this envelope after you visit the notifications list.

I think that functions to delete items in the notifications list makes users think that they must do some work there and make it appear more complicated, so I'd like to try implementing this a bit differently.

On Wed, Jul 9, 2014 at 11:41 AM, Nikolaus Demmel notifications@github.com wrote:

I'm not sure what you mean by "red alert"? I assume you are referring to the red envelope indicating new forum responses. AFAIR that alert used to always clear as soon as you looked at your responses. However I did like the ability to delete selective response items from the inbox to remind me where I haven't responded immediatly and would like to maybe do so later on.

Just to be clear, this is how my inbox looks: https://www.dropbox.com/s/vo306sk1fb1qr0a/Screenshot%202014-07-09%2016.40.23.png

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-48480044 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[evgenyfadeev]

The layout will also be cleaned up.

On Wed, Jul 9, 2014 at 11:44 AM, Evgeny Fadeev evgeny.fadeev@gmail.com wrote:

Yes, the red flag. I'll need to do some work to auto-clear this envelope after you visit the notifications list.

I think that functions to delete items in the notifications list makes users think that they must do some work there and make it appear more complicated, so I'd like to try implementing this a bit differently.

On Wed, Jul 9, 2014 at 11:41 AM, Nikolaus Demmel <notifications@github.com

wrote:

I'm not sure what you mean by "red alert"? I assume you are referring to the red envelope indicating new forum responses. AFAIR that alert used to always clear as soon as you looked at your responses. However I did like the ability to delete selective response items from the inbox to remind me where I haven't responded immediatly and would like to maybe do so later on.

Just to be clear, this is how my inbox looks: https://www.dropbox.com/s/vo306sk1fb1qr0a/Screenshot%202014-07-09%2016.40.23.png

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-48480044 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

Askbot Valparaiso, Chile skype: evgeny-fadeev

[NikolausDemmel]

I don't think notification/mail inboxes are that hard to understand for users in general. But I will let you do your thing :-)

Thanks!

[dornhege]

Right now it seems that when I "block user and delete all posts" it will only remove that one post from the moderation queue. The rest of the spam questions stay unless manually removed. I'd suggest removing already deleted questions from the moderation queue.

[evgenyfadeev]

It should also clear other posts by the same user, but not touch posts by other users.

On Wed, Jul 9, 2014 at 12:01 PM, Christian Dornhege < notifications@github.com> wrote:

Right now it seems that when I "block user and delete all posts" it will only remove that one post from the moderation queue. The rest of the spam questions stay unless manually removed. I'd suggest removing already deleted questions from the moderation queue.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-48484564 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[NikolausDemmel]

The posts get deleted, but it seems they don't disappear from the moderation queue.

[evgenyfadeev]

Yes, I see some of that, thanks.

On Wed, Jul 9, 2014 at 8:11 PM, Nikolaus Demmel notifications@github.com wrote:

The posts get deleted, but it seems they don't disappear from the moderation queue.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-48547008 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[NikolausDemmel]

Are new users supposed to be moderated right now? I had a bunch of bunch of SPAM and a valid question in the moderation queue earlier, but now we just had new spamm users being able to post again.

[evgenyfadeev]

Yes, all posts of "watched" users are on the queue. For those - that are not spam - even if they are perhaps crappy posts approve those users and you won't see them on the queue any more.

That is you have to use more discretion when moderating as some of the posts there are good.

On Thu, Jul 10, 2014 at 6:48 AM, Nikolaus Demmel notifications@github.com wrote:

Are new users supposed to be moderated right now? I had a bunch of bunch of SPAM and a valid question in the moderation queue earlier, but now we just had new spamm users being able to post again.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-48584975 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[dornhege]

There just was spam on the main page that was also in the moderation queue.

The account had 1 karma and was a "registered user".

[dornhege]

Just to be save: What will happen if I check the box when someone "marked offensive" a post and I "delete posts and block users" - will this block the user that posted the spam or the user that flagged it. Currently it shows the name and icon of the flagging user.

[evgenyfadeev]

In that case post is deleted of the user who made it, not the one who flagged.

I will fix the presentation of the samples.

On Thu, Jul 10, 2014 at 7:30 AM, Christian Dornhege < notifications@github.com> wrote:

Just to be save: What will happen if I check the box when someone "marked offensive" a post and I "delete posts and block users" - will this block the user that posted the spam or the user that flagged it. Currently it shows the name and icon of the flagging user.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-48588659 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[evgenyfadeev]

It is not showing whether the user is "approved" or "watched", I will fix this.

On Thu, Jul 10, 2014 at 7:27 AM, Christian Dornhege < notifications@github.com> wrote:

There just was spam on the main page that was also in the moderation queue.

The account had 1 karma and was a "registered user".

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-48588398 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[NikolausDemmel]

Yes, all posts of "watched" users are on the queue. For those - that are not spam - even if they are perhaps crappy posts approve those users and you won't see them on the queue any more.

This is how I expected things to work.

However spam of (some of the) unapproved users still gets through to the main page.

[evgenyfadeev]

This is because we are in the "audit" mode - showing all posts, but moderating them after the fact.

Some more work is needed to implement "premoderation".

Later we'll add statistical classifier of spam.

On Thu, Jul 10, 2014 at 7:37 AM, Nikolaus Demmel notifications@github.com wrote:

Yes, all posts of "watched" users are on the queue. For those - that are not spam - even if they are perhaps crappy posts approve those users and you won't see them on the queue any more.

This is how I expected things to work.

However spam of (some of the) unapproved users still gets through to the main page.

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-48589170 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[NikolausDemmel]

Ah ok, I though "premoderation" was supposed to be in effect already. Fine.

(I'm not sure I understand the point of showing all new users' posts in the moderation queue if they go to main page anyway. I guess it is as easy/fast/slow for moderators to pick out spam from the main page than it is to pick out real questions from the moderation queue, but probably I'm just not getting it. If this is just an intermediary step towards "premoderation" and other techniques anyway, then no further explaining/discussion needed.)

[evgenyfadeev]

The "premoderation" (not enabled on your site yet) kind of works but has an issue that the new posts appear to be "lost" - they are not shown to even the author.

I need to add display of moderated posts only to authors, then we can enable premoderation.

It is not intended to show bad posts on the main page, but is a feature of the method used.

On Thu, Jul 10, 2014 at 7:47 AM, Nikolaus Demmel notifications@github.com wrote:

Ah ok, I though "premoderation" was supposed to be in effect already. Fine.

(I'm not sure I understand the point of showing all new users' posts in the moderation queue if they go to main page anyway. I guess it is as easy/fast/slow for moderators to pick out spam from the main page than it is to pick out real questions from the moderation queue, but probably I'm just not getting it. If this is just an intermediary step towards "premoderation" and other techniques anyway, then no further explaining/discussion needed.)

— Reply to this email directly or view it on GitHub https://github.com/ros-infrastructure/answers.ros.org/issues/76#issuecomment-48589914 .

Askbot Valparaiso, Chile skype: evgeny-fadeev

[NikolausDemmel]

Some more info about the issue raised above, where "block user and delete content" does not clear the entries from the moderation queue, neccessitating repeated handling of the same posts:

Assume that a bunch of posts from a new spam user are in the moderation queue and I select the first post and hit "block users and delete all content", the user and all its posts always get blocked/deleted and all posts usually disappear from the moderation queue. This is just as expected. However, sometimes a few posts remain in the queue and only get removed when you select them and hit "reject for reason SPAM". I'm not sure if this is related to whether the post has been flagged by someone or not.

On the other hand, when I see a bunch of spam posts from a new user and delete them through the moderation page of the user's profile (not through the moderation queue), then these posts seem to remain in the moderation queue (they are correctly deleted and the users blocked, though).

[NikolausDemmel]

It seems that now a lot of posts appear in the moderation queue that shouldn't. E.g. right now there are two posts from "registered users" in the moderation queue. One has 2 posts, one 6.

[tfoote]

We've resolved the major issue by premoderation. I've created some followups for improvements. Closing this ticket.

[gavanderhoorn]

Nothing serious, just an update: seems the spam 'bots' have something new (and I almost fell for it):

john read wrote:

Gary Servin has provided you quite relevant link which contains many informative points regarding your question. ipad applications developer

It's a comment on an actual question (this one), referencing an actual user (that also commented on the same question). Notice the spam link at the end of the (boilerplate) comment.