[ATTACK on parse-server] Is there a way to know if it's count operation request in beforeFind?

[pungme]

Hi there,

We got an attack by creating count request to the largest collection and we would like to prevent this. Is there a way to know if it's a count request? so that we can block it?

Thank you so much in advance!

[pungme]

@flovilmart would be really grateful if you could help. Thanks a lot!

[flovilmart]

You should probably block the IP address of the attacker, you can disable public find (requireAuthentication: true) on the class

[pungme]

Hi!

Thanks a lot for a prompt reply, the problem is, he change the ip every hour or so and about the requiresAuthentication, he also create an account for that. (Yes we block the account but he'll just create a new one)

Is there a way to just block the count operation? we never use it in our client so that's totally fine. if you can point me to where I can do that'd be great.

Thank you so much and hope to hear from you soon

[flovilmart]

Something you can explore is setting the query timeout to an acceptable value (within range of you regular operations) (maxTimeMS) in your parse server constuctor under databaseOptions:

{
   appId: '....',
   databaseOptions: {
      maxTimeMS: 200
   }
}

The queries will timeout / close after 200ms.

Then you should probably inspect the queries in a middleware (before parse-server) and reject the ones that have counts.

const parseServer = new ParseServer();
const app = express();
app.use('/1', function(req, res, next) {
  const isCount = req.query.indexOf('count') > 0; // not tested
  if ( isCount) {  // identify the count query and res.send(401); OR don't call next to timeout the request
  } else  {
    next() // process accordingly
  }
});
app.use('/1', parseServer);

[pungme]

Hi!

Thank you very much for your reply, I really really appreciate your help. We did try that but the req.query is empty. This is how we did it:

  app.use(bodyParser.json());
    app.use(bodyParser.urlencoded({ extended: true }));
    app.post('/classes/LargeClass', function(req, res,next) {
        console.log(req.query) // empty
    });

Any suggestion? Thank you so much!

[pungme]

Just a quick question, is there a way to know it from beforeFind? when I test with the count request, the request inside beforeFind is empty

[flovilmart]

did you setup your middleware before parse-server? Also note that it can go through he POST body if he's using the same vector as the JS-SDK is using. looking there: https://github.com/parse-community/parse-server/blob/master/src/Routers/ClassesRouter.js#L12 it is possible that the count key is in the body.

[flovilmart]

TBH I had other plans for my Saturday evening. We usually don't provide hands on support for those situations. Send away your index.js so I can have a 👀 (with redacted apiKeys/ master/DB etc..) for Florent [at] flovilmart [dot] com

[pungme]

Hey man, Thank you so much!!!! appreciate your help. I've just sent it to you. Cheers!

[flovilmart]

alright, got it!

[flovilmart]

Closing that one for now. Also, on the beforeFind (to address the original question), the query doesn't have a count property on it as it's a transient option, I can add it alongside the request probably.

[pungme]

Hey Florian, That would be really great :-) I'll close the issue since there's already pr for it

[benedickt12]

@pungme

We also suffered from this issue once but we figure out how to solve it permanently without depending on Parse itself

we use parse and we love using it! this is just maybe the costs of using open source, because everybody can learn from source code to find exploits or bugs that bad people can use ... It is very common in the open source world, think about people who were using WordPress or phpBB, it is easy to attack blogs or forums using these softwares.

We are hosting our servers on AWS and were using mLab for our MonogDB instances, we thought to use PostgreSQL backed by AWS RDS, but at the moment parse has poor support to it and still need much work, in addition we will miss the power that MongoDB offers (flexibility, scalability ... and cost effectiveness)

The kind of attack you mentioned is targeting MongoDB, we suffered from this when we were using mLab, cause mLab seems to use default configurations. There were a lot we could do if we had control and full access to our MongoDB instances. Unfortunately mLab support cannot do a lot for you if you weren't using the dedicated plan, which is really expensive!

These Docs helped us to setup our MongoDB and Parse Server correctly to ensure high scalability and availablity https://d0.awsstatic.com/whitepapers/DDoS_White_Paper_June2015.pdf https://d0.awsstatic.com/whitepapers/AWS_NoSQL_MongoDB.pdf

I will summarize what we learnt and did:

Note : this is only relevant if you were using AWS.

---

Route53

We had to move our domains to use the DNS Service offered by AWS which called Route53, a reliable, fast, cost effective, scalable and high available DNS Service that works perfectly with AWS Services

Load Balancing

You must use Load Balancer and deploy your Parse Server on at least 2 EC2 instances, you can also use Auto-Scalling group and let AWS scales your Parse Server up and down based on the load (request/sec, CPU ... etc). Make sure to run Parse Server behind a Reversed Proxy like HAProxy or NGINX, we are using NGINX. It is also good to consider deploying your instances on Private Network.

The easiest way to do this is to use AWS Application ELB with SSL and Health Check enabled. Although I'm not really fun of AWS ELB and prefer to use NGINX also as Load Balancer. If you use AWS ELB, you also need to continue using NGINX as Reversed Proxy, because NGINX handles requests better than just using naked NodeJS.

With NGINX I have full control what it can do and can optimize it based on my needs, but it is really pain in the ass if you are not familiar with it. If you don't want to have much pain, ELB is a good option to go with. It is also easier to use ELB, because when using NGINX you need to maintain multiple instances to work as Load Balancer and use DNS Load Balancing to keep you application high available. ELB is built with High Availability out of the box, so you don't need to care about it.

In case you use NGINX then you need to create an "A" DNS record on your domain with multiple IPs and put the IPs of the EC2 instances, where your NGINX is running. This looks like this:

api.domain.com A 111.111.111.111 222.222.222.222

Setup your Load Balancer in Public Network and let it routes traffics to the instances in Private Network, where Parse Server is running.

Use Load Balancer no matter how big is your App

Replication/Sharding on MongoDB

I talked first about MongoDB at the beginning, because it is really imported because, once your MongoDB instances are slow or partially dead, nothing else can help you to keep your app available and reachable. We were using mLab and we though on using Sharding but this comes with a lot of costs and we also still can't get full control to do optimize MongoDB to our needs.

But why shard ?

It is better to shard at early stage because sharding helps to balance loads among your instances, sometimes your queries can run a bit slower because mongos needs sometimes to query all servers to give you the information you need, but this is actually not an issue, because when data are migrated between shards and chunks during the Balancing window, mongos will know what shard needs to hit with the query next time, so make sure that you also deploy your Config Server as a ReplicaSet on separate Instances! Also make sure to use multiple mongos

Sharding is really useful to minimize the effects of the kind of attacks described in this issue. Although sharding will make count not accurate but an accurate count doesn't hurt as a slow or even unavailable MongoDB.

We had to modify our Parse Server to pass readPreference, so we can keep our app responsive and available during a MongoDB failover (when a primary is not available). With this option we tell Parse Server and MongoDB during a failover to read/query data from a secondary.

Use Network Optimized/Enhanced EC2 Instances

A failover takes usually based on Network, Hardware and your Configurations 30 seconds to 5 Minutes. So if you are deploying MongoDB by yourself on MonogDB make sure to use network optimized instances like M4, C3, R1 ... etc, T2 Instances have really pour performance and shouldn't be used on Production, this applies to your Parse Server deployment too!

use IO Optimized EBS

If you use General Purpose EBS (GP2) with your MongoDB instances you will have bad performance, because you don't have enough IO Throughput. Operations like count, skip and not indexed find/update makes MongoDB scan through the documents and during this time MongoDB starts writing to disk where performance draws back.

This is also one of the reasons why we stopped using mLab because mLab uses GP2 on all its plans except with the Dedicated Cluster High Performance plan, which starts from $1390, We need at least 16GBs for our MongoDB. That comes with a total costs of at least 3 x $3,750 = $11250 / month with mLab.. Doing it by ourselves costs us less than $1,500 / month

Use Placement Groups

Create multiple Placement Groups and deploy your MongoDB on it, you need to create a Placement Group per AZ, because Placement Groups cannot span multiple AZ.

EC2 instances launched in a Placement Group use the highest Network throughput, this is useful for MongoDB replication to prevent replication lag and network latency and help reduce the failover time

You can (but should and must not) also deploy Parse Server to the same Placement Group so it can reach your MongoDB but you need to considers these limitation before doing http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html#concepts-placement-groups It depends on your setup

CAUTION: if this is not well setup, it will work against you!

Use AWS WAF and AWS Shield

At the late beginning we used Cloudflare, it didn't really help us. Whenever we asked for help or support, the support team asked us to upgrade to Business or an Enterprise plan. But we wanted to use AWS Route53 because it integrates well with AWS Services (like ELB, Autoscalling ... etc). So after we setup our domain on Route53 we looked on AWS and found out that we are ready to go without Cloudflare. First AWS Route53 gives you a basic protection against some kind of DDoS Attacks but if you need a better protection, you have to use the AWS WAF and Shield. It works the same like Cloudflare but it is much cheaper and works as you need it.

However you need to configure a lot of things and set rules by yourself. Cloudflare collects logs and learns from attacks from different clients then it auto create rules. This is good if you were using Wordpress or any popular CMS or Software, but if you use your own API or a non-popular software or something like (Parse Server), there are really not many advantages in using Cloudflare. Cloudflare lets you also define up to 25 Rules but only on their expensive Business and Enterprise Plans.

AWS Shield gives you access to 24x7 DDoS Response Team, they are better than Cloudflare Support Team.

use CloudFront

If you have a web application, serve your css/js/images through AWS CloudFront

@flovilmart If you find this reasonable add it to the docs or I can add it too with PR whenever I find time ?

[flovilmart]

Would be nice part of the docs repo, perhaps a new section for mongodb scaling and AWS best practices. Thanks for sharing and pushing Parse-server to the limits :)

[flovilmart]

@benedickt12 Also:

We had to modify our Parse Server to pass readPreference

You can pass the database options right in the parse constructor with databaseOptions: {} this hash is passed to MongoClient.connect

Sharding is really useful to minimize the effects of the kind of attacks

yes but comes at a high cost if querying on non shard key etc... I would not recommend sharding unless you really know what you're doing AND you'll have operational benefits of sharding not only mitigation of that kind of attack :)

[pungme]

@benedickt12

Thank you so much for sharing, that's definitely something we're discussing within the team. We're already using Elastic Beanstalk as a loadbalancer.

The other way is to verify each request and make sure that the request is fall into the option that we predefined. This definitely make it a bit harder for developer cause you can't query freely anymore but ultimately solve all other unwanted query.

[benedickt12]

I'm not sure how feasible it is, to verify all requests. Well in AWS WAF or Cloudflare Business you can setup rules and define requests that will should be blocked/filtered by the service, you don't need to do that on the Parse Server level by adding some middlewares.

But for the case you described your request for count on parse look like this

GET /classes/classA?limit=0&count=1

it is normal request that Parse SDK would call when you do this

classA = new Parse.Query('classA');
classA.count()

Good Luck!.