More intelligent anti flooding measures

luceos commented 8 years ago

Related to the spam flood just now who posted over 20 discussions in a short time frame I'd like to start a discussion how to improve this before releasing a stable version.

Some ideas came to mind:

Fail2Ban

Fail2ban scans log files and based on the repetition of a certain entry will ban an ip based on how often it was found. We could easily check for number of created posts within a specific time frame and do a temporary suspend or more aggressively throttle.

User create time based throttling

We can modify the throttling so that it is more strict for new users. Once you have been active longer on the forums your throttling is more lenient or can completely be ignored.

User flagging

When multiple users flag the same post (or posts from the same user) we can act more aggressively. Eg with multiple posts from the same user, set all to delete and suspend directly.

Approval

We can make the approval extension a bit more advanced. For one it seems it only requires approval for replies, not for discussion creation. Secondly we can auto-mark discussions by users with posts that have been flagged to be approved first (see user flagging above).

Behavior analysis

Some things we can scan to identify malid users:

Same content in following posts.
Interval between posts (especially those timings that circumvent throttling).
New users with the same IP as a suspended/deleted user.

This is a quick draft and really needs a) more discussion and b) action to implement the preventive features through follow-up issues.

luceos commented 8 years ago

@franzliedke also proposed an emergency mode that places the whole forum in read only mode and showing a global message.

dsevillamartin commented 8 years ago

New users with the same IP as a suspended/deleted user.

That could work... but with not much success. There are many free VPNs to hide your IP, and also for s temporary email :disappointed: It will only suspend those who have no clue how to do it at first, but then will see it is very easy to trick Flarum and IP.

luceos commented 8 years ago

@flarum/core this needs attention. I just stopped the second spam run by deleting the user completely. I am moving this to beta 7 if you agree.

Any community should be able to moderate the forum with the smallest number of mods possible. Spam runs complicate this immensily.

luceos commented 8 years ago

A few more thoughts:

allow certain endpoints only with a CSRF token (option in admin)
allow new users no access to endpoints without CSRF token

kulga commented 8 years ago

Modifying the suspension tool:

Below the "Suspension Status", a option to soft-delete posts the user made in the last n hours and a info tip of how many hours since registration.

Basically to allow soft-deleting versus hard delete (by removing a user)

emergency mode that places the whole forum in read only mode and showing a global message.

I really don't like this. It solves the spamming problem but is heavy handed that it affects everybody Stopping everybody with less then n hours since registration, no likes / replies to their account would still let most users continue without issue.

Perhaps check the period between first and last post. I assume spammers will post a whole lot and then leave - forever. If a user posts a month ago and then 5 times in a hour, no match but 10 times in 1 hour and never posted before?

When multiple users flag the same post (or posts from the same user) we can act more aggressively.

This should only apply to certain users for a limited time. This can have a reverse effect of "unpopular" posts being taken down because a lot of people falsely flagged it.

@datitisev It will only suspend those who have no clue how to do it at first, but then will see it is very easy to trick Flarum and IP.

Yes - but If you can't go by username, their ip address or email, what can you go on? You are right however, especially as this relates to a botnet - where every ip / email / username could be different. Which is where "intelligent anti-flooding" comes into play, into a race between detecting and deleting spam, and bots posting in ever more clever methods, trying to look like a normal user.

Z3120FX commented 8 years ago

Just like i pointed out before: https://discuss.flarum.org/d/2575-trust-level

Discourse like trust level system perfectly stops, if not, effectively reduce this kind of spams.

Combined with Akismet, this method is the best way to fight spam. It needs less energy/manpower so Moderators can deal with other important things without putting too much time killing spams.

Reflic commented 8 years ago

I really love that idea of an emercency mode with completly read only access and a nice message on the top of the page.

That would be awesome and gives the admin a little bit of time before finding the source.

johnhearfield commented 8 years ago

Another forum system has a setting Enable Flood Check which basically does this:

Once enabled, users are disallowed from posting within a given time span of their last post (or discussion, or both).
Set a floodcheck timespan of 60 seconds, a user cannot post for 60 seconds after.
Administrators and moderations can be exempt from the floodcheck

Seems a pretty simple solution, but might take some time to implement properly. I think you need to hook into the call which submits the discussion or submits the reply and first does a check from last post

patrickblackjr commented 8 years ago

I was going to suggest what @johnhearfield said, a flooding system where there is a 30 second delay or so before the user can post again would help. Combined with emergency mode, a trusted system, etc. would be nice.

JoshStrobl commented 5 years ago

Background

Solus has been leveraging the Akismet support in Flarum on our Flarum to block repeated Chinese-language SEO spam.

Pain-point

While this support has been fantastic and has reduced our workload substantially when it comes to clearing out discussions, all of this SEO spam comes in the form of one post in one new discussion / topic. It's never being posted in an existing topic.

So while the plugin rightfully flags the post itself and marks it as not approved, if it's the only post in the discussion, it doesn't remove the discussion itself (which can always be restored by an admin should the akismet flag something that is a false positive). This means you'll have loads of topics, all which have a spammy title, with no actual content inside it, and this is all viewable to the user since the discussion itself isn't flagged.

How this could be resolved

I'd love to see the Akismet plugin automatically remove a discussion if the flagged post is the only one in the discussion as part of the "approval" sub-section of this task.

JoshStrobl commented 5 years ago

An example of what we're having to deal with on a daily basis. Cleaning this up is really not fun.

spam

luceos commented 5 years ago

@joshstrobl try the stopforumspam extension.

JoshStrobl commented 5 years ago

@JoshStrobl try the stopforumspam extension.

We're already using that.

dsevillamartin commented 5 years ago

@JoshStrobl What about fof/spamblock ? It automatically suspends the user indefinitely & soft deletes all their discussions & posts (as well as reporting them to StopForumSpam if you have it enabled).

However, we probably want to improve it in core / Akismet as well. You mentioned flooding measures, perhaps having a value configurable per tag and/or per group would help. Having the default time be configurable as well, though it also depends on extensions using that value instead of hardcoding their own in their custom flood handling.

JoshStrobl commented 5 years ago

@JoshStrobl What about fof/spamblock ? It automatically suspends the user indefinitely & soft deletes all their discussions & posts (as well as reporting them to StopForumSpam if you have it enabled).

However, we probably want to improve it in core / Akismet as well. You mentioned flooding measures, perhaps having a value configurable per tag and/or per group would help. Having the default time be configurable as well, though it also depends on extensions using that value instead of hardcoding their own in their custom flood handling.

Thank you for the recommendation, I'll look into enabling the spamblock plugin. However in my opinion, it seems like a lot of this is just reactive (addressing spam which already exists that users will see until a moderator handles it) rather than proactive (preventing or reducing the likelihood of it being posted in the first place). Improving akismet integration to proactively remove the parent post would be a good start, as would flood protection and the ability to add custom security questions during sign-up (outside the scope of this issue though).

askvortsov1 commented 3 years ago

flarum/framework#2170 could be useful as well

flarum / issue-archive